bug: Nvidia NIM setting is not available

[nikiluk]

• I have searched the existing issues

Current behavior

Nvidia NIM setting is not available on the laptop with no Nvidia driver while available on the other with the Nvidia driver.

Minimum reproduction step

1. Go to Settings
2. See that the setting for Nvidia NIM is not available on the laptop that does not have Nvidia GPU card

Expected behavior

Nvidia NIM is used for remote Nvidia API called from the Jan interface. The API calls does not require Nvidia GPU on the client and are not performance-hungry.

Therefore, it's expected to have this setting on any machine regardless of the presence of the Nvidia GPU, in the same way as calls to OpenAI API or Anthropic API.

Screenshots / Logs

Laptop 1 No discrete GPU

Laptop 2 Discrete GPU

Jan version

0.5.2

In which operating systems have you tested?

• macOS
• Windows
• Linux

Environment details

No response

[louis-menlo]

Thanks @marknguyen1302 for adding back the engine support from Cortex

[nikiluk]

@Van-QA does not seem to be affecting the UI

Nvidia settings still not present, Jan 0.5.2.

[Van-QA]

@nikiluk can you try our nightly build? https://github.com/janhq/jan?tab=readme-ov-file#download

[nikiluk]

@Van-QA after reinstalling it to the nightly build I've realized that the new Model settings are not accepting for example the custom URL of the inference API that was previously possible. For example, I've used Microsoft Azure Open API endpoint but now it's not possible to set it in the new UI. Shall I raise the new ticket? Not doing it given I'm now operating in the nightly build.

[Van-QA]

@Van-QA after reinstalling it to the nightly build I've realized that the new Model settings are not accepting for example the custom URL of the inference API that was previously possible. For example, I've used Microsoft Azure Open API endpoint but now it's not possible to set it in the new UI. Shall I raise the new ticket? Not doing it given I'm now operating in the nightly build.

May I ask, how did you usually config the custom Url in Jan? Is it via the Jan settings? If so, you can go to the Jan data folder /models folder, and clone/modify the yaml file of the model, you can take the gpt-4.yaml as an example 🙏

[nikiluk]

@Van-QA yes, via Jan settings UI page

[Van-QA]

hi @nikiluk, to workaround for the time being, please temporally modify the url in yaml.

We will resolve the issue via #3341

[nikiluk]

@Van-QA thank you, will try that