Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Integration of Wav-2-Lip & Talking-Face To Give Leon A "Physical" Form #526

Open
sarutobiumon opened this issue May 30, 2024 · 0 comments
Open
Labels
feature request Indicates new feature requests.

Comments

@sarutobiumon
Copy link

sarutobiumon commented May 30, 2024

Feature Use Case

Use case is to give Leon a personality that is not just a voice, but an actual visible form that can be interacted with like a virtual person or anime character, a fantasy based completely imaginary character...etc

Feature Proposal

Using an animated gif image, incorporate the features of Wav-2-Lip below into the main Leon App:
https://github.com/anothermartz/Easy-Wav2Lip/releases/tag/v8.3_release
Text that is generated by Leon can be either spoken in audio only, or recorded as a wav file and then animated live via the "simulated" talking-face from the GIF image that is automatically animated by the wav-2-lip model.

Additionally, RAG can be used to give the personality specific lines to re-use as part of their core personality.

If real time TTS is needed for this to work, please consider Piper as it also supports multiple languages and is super easy to train on own machine, can run on super slow and weak PC's on CPU and still does near-real-time generation.

Thanks for the awesome work!

@sarutobiumon sarutobiumon added the feature request Indicates new feature requests. label May 30, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request Indicates new feature requests.
Projects
None yet
Development

No branches or pull requests

1 participant