---
name: lip-sync
title: Lip-sync that holds
summary: Plan A for singing characters — drive the mouth from the voice instead of fixing it afterwards.
tags: [video, music, lip-sync]
---
# Lip-sync that holds (Plan A)

Fixing mouths after generation is slow and rarely works: a model that generated a wide-open shouting mouth cannot be "edited" into words. Drive the mouth from the start.

1. **Isolate the lead vocal.** Separate the song into lead vocal and the rest; remove backing vocals and harmonies. The mouth should follow one voice.
2. **Cut it into lines.** 3–8 seconds per segment, starting and ending in breaths.
3. **Front-facing close-ups only.** Chest-up, whole head in frame, mouth closed or slightly open in the start frame, locked camera.
4. **Voice-driven generation.** Feed the start frame and the vocal segment to an audio-driven avatar model. Prompt: the scene, plus "mouth closed when there is no voice".
5. **Everything else is not a mouth shot.** Wides, backs, dancing, hands, props. Mark in the shot list which lines show the singing face; design the rest to avoid it.
6. **Measure, do not eyeball.** Track mouth opening per frame and compare with the vocal loudness curve. Low correlation or an open mouth during silence = redo that segment only.

Plan B for the hardest fast rap: a person performs the lines to camera and a motion/expression transfer tool maps the performance onto the character (the performer's face never appears).