epic: cortex engines commands

[vansangpfiev]

Goal

• Cortex can install engines (e.g. llama.cpp, tensorrt-llm)
• Strong error recovery

Spec

• https://github.com/janhq/cortex/issues/1046

Tasklist

Install (this issue)

• HTTP API support for cortex engines install
• CLI support for cortex engines <name (cortex.llamacpp/cortex.tensorrt-llm/cortex.onnx)> install
• Downloading and unzip engine file (currently store engine to engines/<engine>)
• Downloading and unzip cuda toolkit (if needed - because, e.g. macos does not need this)
• Libcurl adding libcurl to support pause/resume download #1179
• Hardware detection
  • AVX information
  • OS
  • CPU arch
  • GPU driver version

Engines list

• feat: cortex engines list #1074

Engines get

• feat: cortex engines get <engine_id> #1073

Uninstall

• feat: cortex engines uninstall <engine_id>  #1088

Dependency Packing

• CI to pack CUDA dependencies for cortex.llamacpp #1085
• feat: Set dll libraries search path for Cortex on Windows #1047

Outstanding / Bugs:

• epic: Semantic naming for Cortex Engines? #1168

Some resources

Nvidia Support Matrix
CUDA Compatibility

[namchuai]

@vansangpfiev @nguyenhoangthuan99 , please add if I missing any. Thank you 🙏

[dan-homebrew]

@namchuai Can I clarify a few things:

• CLI command: is it install or init? If install, let's change this issue name
• Will we do an uninstall command as well?

[dan-homebrew]

@namchuai Another thought is, should the command be:

cortex engines install tensorrt-llm
or 
cortex engines tensorrt-llm install

I had originally gone with cortex engines tensorrt-llm install, because I initially thought that each of the engines would eventually expose its own CLI commands, e.g:

cortex engines tensorrt-llm settings ...
cortex engines tensorrt-llm gpu 2

However, I realize that install is an action on engines, rather than tensorrt-llm (or engine name). In this case, the correct approach should be:

cortex engines install tensorrt-llm
cortex engines uninstall tensorrt-llm
cortex engines list

[namchuai]

@namchuai Can I clarify a few things:

• CLI command: is it install or init? If install, let's change this issue name
• Will we do an uninstall command as well?

1. Yes, it's install. I updated the title.
2. I agree about uninstall
   cc @vansangpfiev @nguyenhoangthuan99 , what do you think?

[namchuai]

@namchuai Another thought is, should the command be:

cortex engines install tensorrt-llm
or 
cortex engines tensorrt-llm install

I had originally gone with cortex engines tensorrt-llm install, because I initially thought that each of the engines would eventually expose its own CLI commands, e.g:

cortex engines tensorrt-llm settings ...
cortex engines tensorrt-llm gpu 2

However, I realize that install is an action on engines, rather than tensorrt-llm (or engine name). In this case, the correct approach should be:

cortex engines install tensorrt-llm
cortex engines uninstall tensorrt-llm
cortex engines list

IMHO, 2nd option is more natural to me.

$ cortex engines install tensorrt-llm

And I think we should support options if any, for example:

$ cortex engines install cortex.llamacpp --vulkan 1

[vansangpfiev]

@namchuai Can I clarify a few things:

• CLI command: is it install or init? If install, let's change this issue name
• Will we do an uninstall command as well?

1. Yes, it's install. I updated the title.
2. I agree about uninstall
   cc @vansangpfiev @nguyenhoangthuan99 , what do you think?

I agree, we should support uninstall. @namchuai Can you please create a ticket for it?

[gabrielle-ong]

Closing this Epic 🌮 🎉 Thanks @vansangpfiev @namchuai!