How to Build a YouTube Shorts Generator with AI

been running a full pipeline from script to finished short for a few months now and it’s down to about 10 minutes per video once everything is dialed in. i know that sounds like cope but the pieces actually fit together now in a way they didn’t even 6 months ago. gonna lay out the whole thing since a few people asked.

## the basic pipeline

the flow i use goes roughly like this:

1. feed a topic or keyword into a text generation model to get a 60-90 second script – tight, punchy, hooks in the first 3 seconds
2. run that script through a voice synthesis model to get the audio track
3. generate or pull b-roll visuals – either through an image/video generation model or a stock asset pipeline
4. auto-caption the audio using a transcription model, format for vertical
5. stitch everything with a lightweight video composition script
6. output a 1080×1920 file ready to upload

the whole thing runs locally or through api calls depending on what you have. i use a mix of both. the composition step is the only part that takes real compute time locally – everything else is api latency.

## temporal consistency – why it’s mostly not a nightmare anymore

this was the thing killing short-form AI video 6 months ago. you’d generate a character or a visual theme and by clip 3 it looked like a completely different subject. the gotchas were everywhere:

– character faces drifting between cuts
– lighting logic breaking mid-video
– text overlays clipping or reflowing inconsistently across frames

the newer generation of video models handles this way better through attention mechanisms that preserve visual context across a longer generation window. it’s not perfect – you still get drift on very specific details like hand positions or small text – but for the typical talking head or b-roll style short it’s workable. my fix for the remaining drift is to keep generated clips under 3 seconds each and treat every cut as a hard scene boundary rather than trying to extend a single generation. that constraint actually makes the edits feel tighter anyway.

## where people get tripped up

the pipeline sounds simple but there are a few places i’ve seen people waste days:

– **audio sync** – if your voice synthesis model returns variable-length audio and you’re timing b-roll to word count, you’ll get drift. sync to the actual audio file duration, not an estimate
– **caption timing** – auto-transcription timestamps are usually off by 50-150ms. fine for long form, visible on short form. add a small negative offset
– **aspect ratio assumptions** – some composition libraries default to 16:9 internally even when you set 9:16. always verify the output dimensions before you run a batch
– **prompt bleed on visuals** – if you’re generating images from script lines directly, adjacent lines that are topically similar will produce near-identical frames. manually vary the visual prompts even if the script doesn’t change much

the batch processing piece is where the 10 minute number comes from. first video in a session takes longer because you’re validating each step. once you’ve confirmed the pipeline output looks right you can queue up 5-10 videos and walk away.

i do want to push back slightly on the idea that the temporal consistency problem is fully solved – it’s solved enough to ship, which is different. for anything requiring a persistent character across a whole series you still need extra work. but for one-off informational shorts or trend content it’s genuinely good enough now.

anyone else running a similar setup – curious what your composition layer looks like and whether you’re doing anything clever with the caption styling step.