Character AI - New TTS system

[sivertheisholt]

They have added a new TTS system, where you can create custom voices or use other voices/predefined voices.
Would be cool if we could implement this. The current implementation is only able to fetch and use the old voices.

https://blog.character.ai/character-voice-for-everyone/

Any thoughts on this? I will take a look at it and see how it works.

[realcoloride]

This is honestly super uncanny and scary but how does the format work? is it a file? is it streamed audio?

[sivertheisholt]

This is honestly super uncanny and scary but how does the format work? is it a file? is it streamed audio?

From what I have tried you can upload an audio file with 10-15 seconds of talking. It is scary accurate...

[realcoloride]

This is honestly super uncanny and scary but how does the format work? is it a file? is it streamed audio?

From what I have tried you can upload an audio file with 10-15 seconds of talking. It is scary accurate...

The google funding must have been crazy but hell no I am not uploading any audio 😭

[IRON-M4N]

So, I cloned the voice of Furina (a character from Genshin Impact) using this audio sample: https://i.imgur.com/Wb9XuG3.mp4

I tried c.ai and play.ht for comparison. The character AI cloning was way faster and sometimes sounded identical to the real one.

Play.ht cloning is really good but sometimes it glitches the voice.

I think it would be a cool feature to use the cloned voices in the module.

[realcoloride]

Will look into it

[realcoloride]

Ok so I have no idea what the format of the data is, but the domain is used in joinOrCreateSession to create an rtc session.

Now i'd actually be surprised if it REALLY opened a webrtc session to do this, but i'll investigate more

for fetching the voices it should be easy, they have a domain for it.

[Jatinverma0786]

Any update brother about voice system?

[realcoloride]

Hello,

Not really. I haven't looked for it since but if I have interest and this is a feature a lot of developers would like to see on the package feel free to let me know.

Cheers

[Jatinverma0786]

Hello,

Not really. I haven't looked for it since but if I have interest and this is a feature a lot of developers would like to see on the package feel free to let me know.

Cheers

I think this feature will be the most demanding feature as I am also a developer and I need this feature so hardly so I request you to work on it please

[wolfboss356]

I also request that you to work on it. I'm an animatronic developer, and using premade Character AI TTS would save me a lot of time wrangling Mozilla TTS and installing a custom dataset on it.

[realcoloride]

After investigation, implementing this feature would mean to rework the entire endpoints to switch to the new (neo) endpoints and switch to a websocket etc.

[Jatinverma0786]

After investigation, implementing this feature would mean to rework the entire endpoints to switch to the new (neo) endpoints and switch to a websocket etc.

So you will work on it or not brother or you need help in coding part?

[realcoloride]

I think I might have to consider both. I am considering rewriting and upgrading to the new API. Will let you know, I am very busy right now.

Cheers :)

[keyserjaya]

Omg, didn't expect it using livekit. It likes we are really chatting with someone :D
Maybe you have some clues to share to make it works?

Maybe some of us can help.

[realcoloride]

Hey there, this should help

However switching to the new endpoints require a total rewrite of the client (which I started doing to support all the new endpoints) but this is much more work needed than I thought & my attempts at handling the websocket part confused me a lot

[Jatinverma0786]

I have foundout that Character.AI is using EdgeTTS LOL

[yukiarimo]

Any updates?

[realcoloride]

Check out #180. I will get to it soon, I am currently busy working on another project.

[realcoloride]

Hey there, sorry for the lack of updates.

Could using the livekit client be a great idea to handle the actual TTS part? Also, upon investigating the way it authorizes access to the actual livekit session is weird.

[Jatinverma0786]

Hey there, sorry for the lack of updates.

Could using the livekit client be a great idea to handle the actual TTS part? Also, upon investigating the way it authorizes access to the actual livekit session is weird.

As for my information it's not using livekit anymore they are using there own system as of now they are fetching Mp3 file from there server.

[realcoloride]

Hello. This is being worked on so all future discussion will be redirected here #180
If you have any problems, feel free to re-open a new issue.