[Feature Request] Simplified API for Inference and HTTP Server Integration

[conorwsullivan]

First I want to express my deep gratitude for this project, thank you guys so much!

I'm writing to inquire about potential improvements to the API for inference, as well as the possibility of integrating an HTTP server for serving text generation requests. Specifically, I'm interested in the following:

1. A simplified and more flexible method for inference that allows for easier integration with external applications. I'm looking to manage chat history in a separate application and would like to have a straightforward way to perform inference on user-provided text.

2. The ability to serve text generation requests over HTTP. I'm interested in implementing a client-server architecture and would like to know if there are plans to include an HTTP server in the repository.

I understand that the repository is rapidly evolving, and I'm excited to see the new features and improvements you have planned. I'm planning to hack an http server together by myself, but I want to find out what your roadmap is so there is not any painful merges for me in the future. I'm open to contributing to the project if there's an opportunity for collaboration.

Could you please share any insights into the roadmap for these features, or let me know if there are any ongoing discussions or pull requests related to them?

[gjmulder]

Please review and use the issue template before submitting new issues

A quick search for "HTTP" and "API" finds #278

[FreedomCoder-dev]

I think, maybe it would be great for this project to act like a inference framework allowing to manage sessions, models and computing over them while providing handy bindings to Golang (or Java?) allowing development of an actual service which handles HTTP traffic with handy development tools

[conorwsullivan]

@gjmulder blame github. I searched for http on the issues and on the discussion, but not on pull requests. Also github by default only wants to search on open issues, when I find that the majority of the time the issues that I'm looking for are already closed, since they are about project direction and how to solve problems that the maintainers already know how to solve. I wish github had a more useful search.