Cortex.cpp: Active Hardware Selection

[dan-menlo]

Goal

• User can select what hardware they want to run Models on
• User can choose CPU
• User can choose which GPU they want to activate
• User can see a warning if GPU is missing dependencies (e.g. CUDA Toolkit)

Key Questions

• What should be the API or
• Engines (e.g. llama.cpp) should inherit the Hardware Selection, and make engine-level changes (e.g. ngl in llama.cpp)

Related

• Cortex.cpp: Hardware Detection #1111
• https://github.com/janhq/jan/issues/3355
• feat: Automatic fallback to CPU in case of GPU loading failure #805

[dan-menlo]

My ideal Jan UX is to allow users to select Hardware at a global level, which then gets passed down to the model's engine at inference time:

• Cortex.cpp will need to have an API (and possibly CLI) for user to define active hardware

cortex hardware list
cortex hardware activate gpu1
cortex hardware deactivate gpu1