lip sync

[m172021]

Apologies if this is off topic.

With rhasppy/piper, we can already do tts -> rten -> in browser.

Are you aware of any Rust package that does lip sync? I.e.

input: image, text
rhasppy/piper/rten => generates wav/pcm data on the fly
then, taking all of the above into account, we want to lip sync the image w/ the audio

Thanks!

[robertknight]

I don't know of a Rust package that can handle the whole process of creating a lip-synced image or video, but you might be able to export a model like Wav2Lip or LipGAN to ONNX (or find an existing ONNX version). You can then convert and run that using rten or one of the other Rust packages that can import ONNX models (ort, Candle, Burn etc.)