Welcome to the Panopto Community

Please note: All new registrants to the Panopto Community Forum must be approved by a forum moderator or admin. As such, if you navigate to a feature that is members-only, you may receive an error page if your registration has not yet been approved. We apologize for any inconvenience and are approving new members as quickly as possible.

Panopto Captioning ASR - machine generated captioning feedback

JAMES HARTYJAMES HARTY Crackerjack
in Use Cases

I'm the IT Administrator for Panopto in Trinity College Dublin, on the emerald isle of Ireland.

Our Student Disability Dept have been in touch seeking to have captioning turned on for asr across all recordings, we are currently in the IT dept doing our due diligence on this topic and looking at all sorts of options, we already have 11,000 recordings and have concerns about turning on asr across the board. We may try doing a pilot with a few depts./recording folders. Nicolas our success a/c mgr has told us that asr improved with new version in March 2020.

What I'd like to know is what universities have turned on asr completely (non paid for service, free asr) on their university and what type of support calls or issues arose?

How did others approach this topic, all ideas and suggestions welcome to give me some more insight?

So far, it doesnt appear to me that many universities have turned on asr across their whole panopto site recordings or maybe they have, have tried panopto slack forum already; not a lot of feedback coming back from other universities, would be interested in anyone who has turned on asr across the board or in phases to hear their experiences

Interesting note below, might be useful to others:

While I'm not an expert on asr accuracy, one 45 min recording I did analysis in some detail recently, with my own simple measuring method of counting the number of sentence sequences and then counting what textual errors were in the sentences, I got a figure of 88% accuracy, mainly the errors were around names. I would suspect that the asr is even more accurate up into the 90%s. There was a whole section of about 20 minutes for the second speaker on the recording where there were pretty much no errors which was impressive, an American accent speaker. Whether the fast talking Irish accent with local dialects will fare aswell no one knows!!!!

Answers

  • Michael EspeyMichael Espey Superstar

    I am the service lead for Panopto here at the University of Iowa (USA), we have ASR available for all Panopto sessions, but the content owner has to go in to request them once available.

    We have been advertising that this is available in Panopto for ~6 months and so far we have had about 1,000 sessions have the ASR captions enabled. We are currently sitting at about 140,000 total sessions in Panopto and there were around 30,000 sessions recorded in that time. Of the ones that I have audited, most have actually gone in and edited the captions. Our worry with this was that people would leave them as they were out of the box, with the added step of the request, the results return in the same time frame, but the owner has a chance to look things over before they go live.

    In my testing, if I am mic'd well, the accuracy is well over 90% since the vendor change. We have noticed some short videos we have published where it hit 100%. I am happy with how the new vendor breaks lines. The turnaround time is also much faster. It isn't perfect, but for ASR, it is a great place for people to start. Editing the captions isn't the easiest thing, but it is there.

    We also have Kaltura available for video, and on Kaltura we opted to have everything get ASR captions automatically. Just as we expected almost none of those have had the ASR captions amended. Some edit everything to 100%, but he accuracy is not quite there for where I would be happy if they are left out of the box. We were OK with this on Kaltura because we use Kaltura for general video things where Panopto is mainly our lecture capture solution.

    There are two things that I want out of Panopto on the subject. I wish that Panopto would allow us to add a dictionary of words. I hate that I need to correct "UICapture" and "Panopto" quite frequently. Outside of that, being able to request ASR in other languages would be amazing. We have a good bit of content in Spanish, and on Panopto they are locked into doing it by hand, paying for professional captions, or using Kaltura to get the ASR captions.

    Hopefully that helps, but let me know if there are any questions I can answer.

    -Michael

  • JAMES HARTYJAMES HARTY Crackerjack

    Michael,

    Greetings from the land of the Emerald Isle, that is Ireland. A Cead a Mile Failte (one hundred thousand welcomes in Gaeilge, our native language, it don't get much better than that!)

    Thanks for taking the time to give detailed information, it is quite useful to hear your scenario.

    I found it quite interesting that the voluntary uptake is quite low by lecturers to import the captions on behalf od the students, I'm not surprised by that, as lecturers probably see it as extra work to have to edit the captions. Good to see some stats on a live university example.

    I'd agree that the machine engine ASR that Panopto introduced in March 2020 seems to have enhanced the accuracy of recordings up to to the high 80s and into the 90%s.

    I believe Panopto are working on more Multilanguage captioning support, in the most recent 9.0 cloud release on 27th June, Italian language support was added for captions if my memory serves me right.

    We are looking at all the options and in particular wondering about whether to turn on autocaptioning ASR for any new recording across our site, so that any new recording will show ASR captions by default, no need for the end user to import captions option. We are concerned about what queries might arise by doing this and what confusion will arise for lecturers who will wonder why some of their older recordings don't have captions while a new one does.

    What is the main reason why you have decided not to turn on auto ASR for all recordings (meaning that new recordings are auto captioned with no lecturer intervention?

    If I can return the favour if you have any queries into the future on what we are doing with Panopto in Trinity College Dublin, Irel;and, feel free to get in touch.

    Slan go foil (goodbye for now - as Gaeilge),

    James, IT Services, TCD

  • JAMES HARTYJAMES HARTY Crackerjack

    I'd agree with you about the dictionary. With the dits on textual inaccuracies, a lot of times, its the same words, it would be nice if there was a search and replace function to speed up corrections like you have in Ms Word.

Sign In or Register to comment.