Using AI to caption Korean e-sports and commentary

malathion

United States361 Posts

November 14 2023 20:27 GMT

ChatGPT isn't the only amazing tool OpenAI has published recently. Did you know about Whisper?

I saw that the whisper large-v2 model is multi-lingual, so that gave me the idea to try using it to translate commentary for Korean webcasts. So, I made a clip from an old SC:BW VOD to test it out, and here's what I came up with using Whisper and then ChatGPT to clean up the results: https://streamable.com/jp0pue

I don't speak Korean so I can't evaluate the quality or accuracy of the translation. If there's anyone here who speaks Korean well, I'd love to know what you think of this.

Apart from the accuracy, I have to say that the process was quite time-consuming. As mentioned in the README, whisper is not very fast at all yet, especially the large-* models. But if someone out there is willing to put in more time, maybe it's now possible to caption more of these videos without actually speaking the language?

GTR

51450 Posts

November 14 2023 21:16 GMT

machine translation for korean, while better than it was 5-10 years ago, still is a bit off at times and it's evident here.

so no, you can't subtitle korean clips with ai - unless you have some grasp of the language and are able to edit the initial machine translation provided (which in that case, you're better off doing the translation normally)

the only benefit i can see this used is to provide the timings for subtitles and to for a translator to work from there

malathion

United States361 Posts

November 15 2023 03:45 GMT

Thanks for the feedback. Can you say more? I noticed that it confuses the loanwords, like "Arbiter" -> "Avatar" or "Psi storm" -> "Siren stop"... is that what you mean by "a bit off at times"?

malathion

United States361 Posts

November 16 2023 17:32 GMT

To be clear, I don't think AI is good enough to beat a human who speaks both languages fluently. But is it good enough to be better than nothing for those of us who don't? If yes, why not do it?

yubo56

688 Posts

November 16 2023 18:34 GMT

This does seem better than nothing. I remembered watching some of TY's post-GSL reviews by using Youtube's "auto caption" and "auto translate" features together (before helpful reddit translators could get around to posting their very good translations), and I was lucky to get more than a word every ten seconds hahaha, this could be a sizable improvement.

It gets especially bad when the background cast audio gets mixed in with the foreground conversation, but that's probably always gonna be hard.

swarminfestor

Malaysia2449 Posts

November 16 2023 18:41 GMT

On November 17 2023 03:34 yubo56 wrote:
This does seem better than nothing. I remembered watching some of TY's post-GSL reviews by using Youtube's "auto caption" and "auto translate" features together (before helpful reddit translators could get around to posting their very good translations), and I was lucky to get more than a word every ten seconds hahaha, this could be a sizable improvement.

It gets especially bad when the background cast audio gets mixed in with the foreground conversation, but that's probably always gonna be hard.

I agreed.

Please or register to reply.

Using AI to caption Korean e-sports and commentary

Completed

Ongoing

Upcoming