bug: Cortex exited with code null immediately after loading a model

[mshpp]

• I have searched the existing issues

Current behavior

Trying to load even the TinyLLaMa Chat 1.1B model doesn't work, Cortex seems to crash immediately after loading the model. This occurs on a fresh AppImage under Fedora 40.

Minimum reproduction step

1. Open Jan
2. Download TinyLLaMa when the app prompts you to do so
3. Write something as the input
4. Press enter

Expected behavior

The model should load and run without a problem.

Screenshots / Logs

2024-06-22T20:03:36.375Z [CORTEX]::CPU information - 2
2024-06-22T20:03:36.377Z [CORTEX]::Debug: Request to kill cortex
2024-06-22T20:03:36.429Z [CORTEX]::Debug: cortex process is terminated
2024-06-22T20:03:36.430Z [CORTEX]::Debug: Spawning cortex subprocess...
2024-06-22T20:03:36.431Z [CORTEX]::Debug: Spawn cortex at path: /home/user/jan/extensions/@janhq/inference-cortex-extension/dist/bin/linux-cpu/cortex-cpp, and args: 1,127.0.0.1,3928
2024-06-22T20:03:36.432Z [APP]::/home/user/jan/extensions/@janhq/inference-cortex-extension/dist/bin/linux-cpu
2024-06-22T20:03:36.550Z [CORTEX]::Debug: cortex is ready
2024-06-22T20:03:36.551Z [CORTEX]::Debug: Loading model with params {"cpu_threads":2,"ctx_len":2048,"prompt_template":"<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>","llama_model_path":"/home/user/jan/models/tinyllama-1.1b/tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf","ngl":23,"system_prompt":"<|system|>\n","user_prompt":"<|user|>\n","ai_prompt":"<|assistant|>","model":"tinyllama-1.1b"}
2024-06-22T20:03:36.746Z [CORTEX]::Debug: cortex exited with code: null
2024-06-22T20:03:37.661Z [CORTEX]::Error: Load model failed with error TypeError: fetch failed
2024-06-22T20:03:37.661Z [CORTEX]::Error: TypeError: fetch failed

Jan version

0.5.1

In which operating systems have you tested?

• macOS
• Windows
• Linux

Environment details

Operating System: Fedora 40
Processor: Intel Core i5-3320M, 2C/4T
RAM: 16Gb

[vansangpfiev]

Thank you for reporting the issue.
Could you please provide more information, it would help us to debug:

• output for cat /proc/cpuinfo
• attach the app.log, it is in ~/jan/logs/ directory

[mshpp]

Sure, both files are attached
cpuinfo.txt
app.log

[vansangpfiev]

From the log and cpuinfo, I think Cortex crashed because we don't have avx build with fma flag off.
Could you please download a nightly build for cortex.llamacpp and replace the .so lib in the path: /home/user/jan/extensions/@janhq/inference-cortex-extension/dist/bin/linux-cpu/cortex-cpp/engines/cortex.llamacpp/
Download link:

https://github.com/janhq/cortex.llamacpp/releases/download/v0.1.18-25.06.24/cortex.llamacpp-0.1.18-25.06.24-linux-amd64-noavx.tar.gz

[mshpp]

This works, now I can get a response from the model. However, it seems that only the first round of inference works -- that is, I can only reliably get a single answer. The next one gets stuck on loading for a long time, and then it either completes or it doesn't and keeps loading. This is with the same TinyLLaMA model, which runs quite fast even on my hardware.

[vansangpfiev]

Thanks for trying the nightly build. Could you please share the app.log again?

[dan-homebrew]

Potentially linked to: #1144?

(Unlikely: user was able to have 1 round of inference work)

[dan-homebrew]

@mshpp I am closing this issue, as we are moving towards a cortex.cpp which is in full C++. The JS version had too many stability issues (like this).

I will be adding this to our Test Checklist for the C++ v0.1. We'll reach out again for you to test v0.1 when we release it (ETA 2 weeks)

[dan-homebrew]

Tracked in #1147