Remote Recorder heartbeat (decrease time it takes to start a AdHoc recording)
Hi,
We need to be able to decrease the time (Syncing) it takes for a Remote Recorder to start after a recording is initiated by API or from the Panopto Remote Recorders Page. Currently it can take from 15 seconds to 3+ minutes after a recording is initialized, untill the recording actually starts on a RemoteRecorder. The time depends on how long since the last time the recorder was used.
From what we can find the default settings currently is as follows: Caffeinated (in active use) -> Awake (near active use or just after active use) -> Napping (wont or hasn't been used in a while; ping times 60s) -> Deep Sleep (hasn't been used in over 2 weeks, ping times 300s*)
As an example we have just released a Crestron integration where Teachers can scan their Access cards on a RFID reader (Mifare) on the lectern, choose a Panopto folder and then start a AdHoc/Live session in Panopto. In this case we optimally would like the recorder to start within 5 seconds, and never over 10. Currently it can take 3+ minutes before the recording starts, but then the teacher has already tried to restart the system sevral times ;)
Please consider to have a more "Live" connection with the RemoteRecorders, so that we can have a more professional experience for the teachers :) We also want to use this integration in our Recording studio, but the Panopto recordings have to start much faster than they currently do.
Comments
Wow. I don't think I've ever seen more then 5-10 seconds for any action taken on the Remote Recorders Page (Start, Stop, Pause) to take effect on the actual box. (I have no experience using the API to control a Remote Recorder.)
I wonder what the tracert and ping times look like between the devices making the calls, the Panopto server, and the Remote Recorder?
What kind of device is the recorder? A Windows box or an off the shelf product like one made by Matrox, Extron, or Epiphan?
I feel like there used to be a way to communicate directly with a Recorder rather than going through the website. Possibly by using something like the serial controls in the Panopto Recorder Light Button Service?
I think there is also a Crestron module somewhere which offers Panopto Remote Recorder control but I don't recall whether it did so via the Web and the Remote Recorders Page, or by communicating directly with the Remote Recorder itself.
The AV team at UNLV demonstrated their use of Crestron controls back in 2019 at the North America Panopto conference in Seattle.
Hi, we are using Matrox Maevex, Epiphan Pearl mini and a few PC based recorders. The issue seems to be the same on all of them. Also the times seems to be the same both from API and from the Remote Recorder page.
Here in an example of a Remote Recorder taking 2.5 minutes to start a Recording. This happens when a recorder has not been used for over 14 days. https://ntnu.cloud.panopto.eu/Panopto/Pages/Viewer.aspx?id=be444300-1cb8-47c0-923a-ae7c008c380a
Off course most recorders is used on a daily or at least weekly basis, but the main problem is when a recorder goes into the "napping" state after some days where it takes 60 seconds to start.
We are basing the Crestron module on the standard Panopto Crestron module, but have made some changes.. The module is the same for all recorders and talks directly with the Panopto API. It is also possible to make a integration directly with the Matrox or Pearl, but since we are using different types we want something universal :)
If a recorder is "awake", how long does it take to start? I wonder if this has to do with differences in the EU and NA cloud infrastructure? The log files on the Remote Recorders might be useful.
Have you opened a case with Panopto support?
My awful idea to ensure no recorders were in a state where they wouldn't be able to respond quickly, is to have the module launch a 1 minute long ad-hoc recording every day at 5 AM.
Not something which should be necessary, but it might improve the situation.
Hi Chaz, the recorder starts in 3 Seconds when it has just been used. The problem is when its been a day or mor since the last time the recorder was used. its the same in SW and HW recorders and its the same if we use RemoteRecorder page in Panopto to start or via API from Crestron.
If a recorder is rebooted, does that decrease the time it takes to start? I'm wondering if scheduling a morning reboot would solve the issue.
I would definitely open a case with support though.
Hi, its confirmed that sleep' state for Remote Recorders kicks in after an extended period without a recording being made.
Feature Request to improve this is now registered as "FR-5679 Improved RR server communication method"
Workaround for us until this is changed is to schedule a dummy recording for 30 sec every weekday morning to keep the AdHock recorders alive. I will update this page once there is any update.
Update:
We have solved this issue with two dirty hacks ;)
01: We now have a system that does "warm up" on all recorders that are used for AdHock Recording. Every morning the recorders is automatically scheduled a 2min recording that is sent to a folder with retention policies. So the recorders now always does start a 2 min dummy recording every night before a weekday, and the recording is then deleted after recording. This keeps the recorders "Alive"
02: We have integrated a API call from the Epiphan Pearl Mini that will pull updated info from Panopto. This curl command will force Pearl to get latest schedule from Panopto.
With these two fixes together rooms with Epiphan Recorders now starts after 3-5 Seconds every time , like it should be :) On Panopto Software recorders or Matrox recorders, we still have up towards 30 seconds before recording starts but never more. Before it could take over 3min.
Hopefully this shows that this can be fixed within Panopto, as its not a very Pro for recordings not to start instantly.
We will release the Crestron Module once it has been field tested.
Warmup:
Start stop AdHock by access card
Very interesting!
Here is a demo of the Integration on our "testbench" We will start rolling out into rooms before Easter :)
https://www.youtube.com/watch?v=MovBGnzUjmE
Yes, I've seen this on PC's here as well when you try to do an ad hoc recording on a remote recorder machine.
Sometimes delays can be up to a couple minutes.
Looks like a great tool! Out of curiosity, how many rooms do you plan to have this installed in?