Python performance is too poor, can it provide an inference library in C++ and provide an OpenAI-compatible API

[geffzhang]

Using bigdl-llm in a production environment, Python performance is too poor, can you provide an inference library in C++ and provide an OpenAI-compatible API

[jason-dai]

Please see https://github.com/intel-analytics/BigDL/tree/main/python/llm/example/CPU/vLLM-Serving

[geffzhang]

This is written in Python, can C++ be added as well? Python's performance is not satisfactory.

[jason-dai]

This is written in Python, can C++ be added as well? Python's performance is not satisfactory.

Unfortunately there is no such plan at this moment