LlamaSpeak cannot run with Llama-3.1-70B-Instruct

[SuWeipeng]

I'm trying to run a 70B model on my Jetson AGX Orin(64x64GB), but it automatically interrupts when I simply replace the 8B model. How can I get the 70B model to run?

When I run the command below, something interrupt the process automatically.

jetson-containers run --env HUGGINGFACE_TOKEN=hf_xxxxx  \
  dustynv/nano_llm:r36.3.0   \
  python3 -m nano_llm.agents.web_chat --api=mlc  --debug   \
    --model meta-llama/Meta-Llama-3.1-70B-Instruct     \
    --asr=whisper --tts=piper

If I run with 8B model, it works very well, for example:

jetson-containers run --env HUGGINGFACE_TOKEN=hf_xxxxx  \
  dustynv/nano_llm:r36.3.0   \
  python3 -m nano_llm.agents.web_chat --api=mlc  --debug   \
    --model meta-llama/Meta-Llama-3.1-8B-Instruct     \
    --asr=whisper --tts=piper

[dusty-nv]

@SuWeipeng can you test Llama-3.1-70B with the baseline nano_llm.chat first? How much memory is it using? I can't recall explicitly testing Llama-3.1-70B, but have done so with Llama-2-70B

[SuWeipeng]

@SuWeipeng can you test Llama-3.1-70B with the baseline nano_llm.chat first? How much memory is it using? I can't recall explicitly testing Llama-3.1-70B, but have done so with Llama-2-70B

@dusty-nv I'm a brand-new man, could you tell me how can I do this?