[G] GenAI subtitles for Korean BW content

The latest Gemini model by Google can handle video input and works surprisingly well for generating English subtitles for Korean Brood War videos. It still makes mistakes here and there and sometimes hallucinates, but it's a big step up from the gibberish you get from YouTube's auto-subtitles. If I had to guesstimate, I’d say it gets >80% right, which feels pretty impressive.

Workflow below.

+ Show Spoiler +

I'm using Gemini 2.5 Pro Preview (05-06) at https://aistudio.google.com/ with default settings. The model is currently free to test. It supports up to 1 million tokens of context; one minute of video is roughly 20k tokens, so the videos above ended up around ~160k–170k tokens each. However this means long content videos like daily proleague or KCM would not work as these exceed the context limit. Maybe chopping them up somehow could work?

Basically, I just pass it the YouTube link and ask it to generate English subtitles.

I've found it works better if I do this in two steps. First, I give it the link and just ask, "what is happening here?".

It will take a while and output a summary.
+ Show Spoiler +

+ Show Spoiler +

Interestingly, this summary often has hallucinations and often doesn’t accurately describe the video. Still, I noticed that when I skip this step and instead ask for subtitles right away, the results are worse. It seems like preloading the context window with Brood War jargon actually helps when it comes time to generate the subtitles. The summary itself being wrong doesn't seem to have any effect on the quality of the subtitles.

After that, I ask it to create the subtitles. The prompt I use looks like this:

+ Show Spoiler +

create english subtitles (.srt)

Quick sanity checklist for SRT files:

Sequential numbers starting at 1.

Timestamp line exactly HH:MM:SS,mmm --> HH:MM:SS,mmm.

The video is less than 1 hour long so all timestamps must start with 00 for HH.

One subtitle text line.

A blank line after every cue.

This should give you subtitles you can copy, save as an .srt file, and use with the video. + Show Spoiler +

The resulting .srt file sometimes has errors which results in missing text; this is often due to the generated formatting being wrong. Most of the times I found it best to just re-run until it worked. Alternatively you could adjust the prompt or fix the .srt yourself. I found the browser addon substital useful, because it allows you to use a local .srt file for youtube videos; and it generated error messages caused by wrong formatting of the .srt files faster than captionify.

I’m still figuring out the best way to share these or upload them for YouTube. I found captionfy, which seems pretty easy to use. You sign up and can create a shareable overlay for any YouTube video. The good thing is that traffic still goes to the original creator, and anyone can upload subtitles that are then available for everyone.

I guess the end goal would be to automate the full pipeline and translate a lot of stuff? It seems captionfy does not have an api so maybe something else might be better suited?

Also the gemini model likely won't be free forever, but with current pricing it should be possible at about ~6cent per 1 minute of content (for videos of similar length) which seems cheap enough? The price scales with (video) input length so longer videos will be more expensive.

Last.Midnight

Australia902 Posts

May 07 2025 01:53 GMT

I was curious about doing this. Surely there are models/n8n setups that can automatically replace/overdub the voice too?

Thanks for sharing man this is great.

Last.Midnight

Australia902 Posts

May 07 2025 02:49 GMT

Recall (https://www.getrecall.ai/) provides written translations and app.vozo.ai apparently does voiceover dubs, but I'm not sure how accurate they are and it's expensive.

Simplistik

1977 Posts

May 07 2025 03:34 GMT

I feel like there is a webservice niche for automating this workflow if anyone has the patience to make to makw it work.

Last.Midnight

Australia902 Posts

May 07 2025 04:24 GMT

yt-dlp for download into ElevenLabs overdub most likely. Only problem is the EL credits.

Possibly with a specialised Eng>Kor model in between.

rtyrt7

46 Posts

May 07 2025 07:34 GMT

Maybe the free models over here would also be helpful, as API:
https://openrouter.ai/models?max_price=0

But it has these limits for the models whose ID is ending in ":free":
- Per-Minute Limit: 20 requests per minute
- Daily Limit: 50 requests per day per account

prosatan

Romania7775 Posts

May 07 2025 07:57 GMT

Thank you Kraekkling !

Kraekkling

Romania383 Posts

May 07 2025 11:56 GMT

On May 07 2025 10:53 Last.Midnight wrote:
Surely there are models/n8n setups that can automatically replace/overdub the voice too?

This is likely not feasible yet. What you're talking about is basically a different piece of technology.

You're right though that there are models that are able to translate audio and output sound in a voice similar to the speaker. However those models are several orders of magnitudes smaller than what we have here and do purely audio-to-audio. They can't handle long-term context. Also there just isn't much training data for these models to be able to properly handle bw jargon.

The advantage of the Gemini model is that we're using information from the video itself (not only the audio) and also tapping inside its "general intelligence" which is due to the very big model size. Additionally here we have inference time scaling, which means the model internally outputs an ensemble of chain-of-thought threads in which it discusses the best way to translate a given passage of video given the overall context, before giving an answer to the user.

However I think we might be not too far away to have models which could do what you suggested, give it 1-2 years at max and we'll be there. The next iteration of openai's omni-series might already do it.

yubo56

687 Posts

May 07 2025 20:31 GMT

On May 07 2025 20:56 Kraekkling wrote:

Show nested quote +

Wait, but you're describing the difficulty of direct audio-audio translation. If you already can do audio -> translated text though, can't you just slap a text-to-speech and have a (basic) audio-audio translation?

I guess you'd have trouble matching the duration of the sentences, but with some simple squeezing and stretching of audio bytes it's still surely quite feasible compared to direct audio-to-audio translation...

prion_

64 Posts

May 07 2025 22:08 GMT

#10

The problem is that it would sound like TikTok caption voice. I mean, not exactly that, but you wouldn't be able to keep the rhythm and modulation of their voices by going audio->text->audio, even if you adjusted for time.

IntoTheWow

is awesome32273 Posts

May 08 2025 02:26 GMT

#11

This is really cool!

Do you think that adding some keywords in the prompt could help the model? Like units, BW jargon, etc? Or are errors due to other factors?

Last.Midnight

Australia902 Posts

May 08 2025 03:36 GMT

#12

I tried ElevenLabs dubbing feature and it works pretty great. Of course I can't speak to the accuracy of the translation but it's certainly more accurate than "translate to English" on Chrome. Only funny thing is that it also dubs the unit sounds so whenever the player isn't speaking he'll repeat SCV commands etc. haha

Lorch

Germany3672 Posts

May 08 2025 13:00 GMT

#13

This is completely useless if you don't speak Korean.
You can never know what part of the translation are accurate and which aren't. You thinking that it sounds reasonable/makes sense is not a great heuristic, especially with how AIs tend to hallucinate.

Would probably need a dedicated bw ai model trained under the supervision of someone who speaks korean + english and is knowledgeable in starcraft to create something worth using.

Kraekkling

Romania383 Posts

May 08 2025 14:32 GMT

#14

On May 08 2025 11:26 IntoTheWow wrote:
This is really cool!

Do you think that adding some keywords in the prompt could help the model? Like units, BW jargon, etc? Or are errors due to other factors?

We're pre-filling the prompt with BW jargon by asking for a video summary first. As to why there are errors - I guess the easiest answer is that the technology is not 100% there yet. Machine translation generally got useful only in the last decade or so... Additionally, BW is a niche domain - one needs a sufficient world model to make sense of the meaning behind words. Koreans often use abbreviations, for example they'd say "zildra" for a zealot/dragoon army; or "sam-hat" (삼햇) for a 3-hatchery opening, etc. I've also tried older models but this one by far is the best one to make sense of stuff like this.

To me, the fact that any of this works at all is pretty crazy.

On May 08 2025 22:00 Lorch wrote:
Would probably need a dedicated bw ai model trained under the supervision of someone who speaks korean + english and is knowledgeable in starcraft to create something worth using.

Unfortunately this won't happen, so for now its either youtube auto-subs or this. + Show Spoiler +

also this is not how models are trained

Maybe someone who speaks Korean could comment? I'm only comparing this to yt auto-subs, and it felt like even with some obvious hallucinations the overall commentary was pretty easy to grasp?

Last.Midnight

Australia902 Posts

May 08 2025 21:24 GMT

#15

On May 08 2025 22:00 Lorch wrote:
This is completely useless if you don't speak Korean.
You can never know what part of the translation are accurate and which aren't. You thinking that it sounds reasonable/makes sense is not a great heuristic, especially with how AIs tend to hallucinate.

Would probably need a dedicated bw ai model trained under the supervision of someone who speaks korean + english and is knowledgeable in starcraft to create something worth using.

Not useless, but not optimal either. Some phrases are lost but things like "focus fire the tank here" when he's also clicking a tank is pretty clear. Hallucinations don't happen as much when models draw from source material, they tend to happen when the trained parameters through a massive database misinterpret a request.

That's why for enterprise integration RAG is all the rage, since the "database" the models link to is the company's data.

Please or register to reply.

[G] GenAI subtitles for Korean BW content

Completed

Ongoing

Upcoming