Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for multiple requests simultaneously #17

Open
WillTDA opened this issue Apr 11, 2023 · 0 comments
Open

Support for multiple requests simultaneously #17

WillTDA opened this issue Apr 11, 2023 · 0 comments

Comments

@WillTDA
Copy link

WillTDA commented Apr 11, 2023

I have created my own expressjs/socket.io web UI using this package. However, this package seems to only allow one connection/request for the model at a time. I could instruct my code to create a new instance of the model every time a user connects to my page, but that would be very inefficient memory-wise.

I had the idea of creating a queue system where the users wait for other requests to complete before serving them, but depending on the length of the answers and how many are waiting, users could be waiting around for a long time.

TL;DR: Would it be possible to allow the package or model to support more than one request/prompt simultaneously without dramatically increasing RAM consumption?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant