Just some general questions

[ZeChArtiahSaher]

I think the setup for the interfaces is great. This thing handles audio, pipes to models and produces rt output. But I was curious if I this project could be pushed also (I can help with what I can) to streamline a couple of things:

• more parallelism or fine graining options?
• some kind of ui overlay perhaps.

Point being I think that if this app can reach the speed or like granularity of something like teams captions experience that would be great. Ofc the target user base doesn't have an nvidia data center and custom fine tuned model, but I think there may be some room for good performance improvements still? What about moving to opencv right? Basically some code could move to c/cpp/cpython and efficient memory allocation. I also thought about what is the degree to which you can parallelize the models.

In terms of ui I think the main improvement here I see is:

• having some kind of way to have the text fixed so eyes don't have to roll every time the new line appears
• maybe it would be nice to have like a UI where when jap characters appear also the english translation appears to, but the translation doesn't bottle neck the transcription if you know what I mean (perhaps may also imply backend changes where captions & translations have split queues if that's not the case)

P.S.

I have good experience with systems dev & programming but not great xp in specifically interfacing with models & various standards around it, however I can get up to speed quickly if there is a clear goal

[ZeChArtiahSaher]

just read some todo list, curious about his one: "Split model up into multiple chunks based on usage"

[cyberofficial]

Hi thanks for the post :)

You are free to fork the project at anytime and contribute anytime, but as for moving projects to different sort of code base, I've decided to stay with Python + Torch defaults, as this keeps the project well to everyone's systems, as other project types do come with various issues or not up to date and relies heavily on third party stuff. I've already given thoughts out trying other code bases to move to. I found my self to keep using Bare python and Torch, though it does leave room for others to port to their own code base if need be. I'm okay with c/cpp stuff, but It's not something i enjoy working with. Also my work flow is already hard coded to make my code/builds easier with other py functions and builds, it's not worth for me to invest in other methods that I have mediocre knowledge in.

As for the UI stuff, I am actively working on a UI system, I've been working with different UI programs and experimenting with options on what will be more better suited, I found a few UIs and learning them, UI is planned for next major.

maybe it would be nice to have like a UI where when jap characters appear also the english translation appears to, but the translation doesn't bottle neck the transcription if you know what I mean (perhaps may also imply backend changes where captions & translations have split queues if that's not the case)

For this, the process happens in order of operation.

The queue is already threaded enough to where everything happens on different threads, but will always catch up with it self in the end and will not trip over it self or skip stuff over, depending on the user settings, stuff may be done slower if record times are higher.

just read some todo list, curious about his one: "Split model up into multiple chunks based on usage"

The models are able to be split up and diced into smaller pieces based on languages. I'm looking at ways to extract the languages out depending on what the user wants, like if they want English -> JP then the model will extract the EN+JP tokens rather keeping other tokens that aren't going to be used. I've gotten a few methods on how, but it's really complicated for the end goal and requires additional programming/scripting setups. That item in the list is near last on my goals of things to do. That ate up too much of my spare time already compared to the other things I want to push out.

So for example if the user does --translation --language Japanese --transcription --target_language Spanish This would make the program spit out English, Japanese, Spanish so then the goal will to be make the model extract those languages out and make a new model file to load, and take up way less vram.