Model inference accuracy

[realhaik]

Running this on RTX4090

C:\AI\Llama2-Onnx\Llama-2-Onnx>python MinimumExample/Example_ONNX_LlamaV2.py --onnx_file 7B_float16/ONNX/LlamaV2_7B_float16.onnx --embedding_file 7B_float16/embeddings.pth --tokenizer_path tokenizer.model --prompt "What is the lightest element?"
C:\Users\Real\AppData\Roaming\Python\Python310\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:69: UserWarning: Specified provider 'DmlExecutionProvider' is not in available provider names.Available providers: 'TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider'
warnings.warn(

What is the lightest element on the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is

C:\AI\Llama2-Onnx\Llama-2-Onnx>
C:\AI\Llama2-Onnx\Llama-2-Onnx>python MinimumExample/Example_ONNX_LlamaV2.py --onnx_file 7B_float16/ONNX/LlamaV2_7B_float16.onnx --embedding_file 7B_float16/embeddings.pth --tokenizer_path tokenizer.model --prompt "What is the lightest element?"
C:\Users\Real\AppData\Roaming\Python\Python310\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:69: UserWarning: Specified provider 'DmlExecutionProvider' is not in available provider names.Available providers: 'TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider'
warnings.warn(

What is the lightest element on the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is the lightest element in the periodic table?
What is the lightest element in the universe?
What is

C:\AI\Llama2-Onnx\Llama-2-Onnx>python MinimumExample/Example_ONNX_LlamaV2.py --onnx_file 7B_float16/ONNX/LlamaV2_7B_float16.onnx --embedding_file 7B_float16/embeddings.pth --tokenizer_path tokenizer.model --prompt "What is the lightest element on the periodic table"
C:\Users\Real\AppData\Roaming\Python\Python310\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:69: UserWarning: Specified provider 'DmlExecutionProvider' is not in available provider names.Available providers: 'TensorrtExecutionProvider, CUDAExecutionProvider, CPUExecutionProvider'
warnings.warn(
?
What is the lightest element on the periodic table? Hydrogen is the lightest element on the periodic table. It is also the most abundant element in the universe. Hydrogen is a gas at room temperature and pressure.
What is the lightest element in the periodic table?
What is the lightest element in the periodic table? Hydrogen is the lightest element in the periodic table. It is also the most abundant element in the universe. Hydrogen is a gas at room temperature and pressure.
What is the lightest element in the periodic table? Hydrogen is the lightest element in the periodic table. It is also the most abundant element in the universe. Hydrogen is a gas at room temperature and pressure.
What is the lightest element in the periodic table? Hydrogen is the lightest element in the periodic table. It is also the most abundant element in the universe. Hydrogen is a gas at room temperature and pressure. Hydrogen is the lightest element in the periodic table. It is also the most abundant element in the universe. Hydrogen is a gas at room temperature and pressure.
What is the lightest element in the periodic

C:\AI\Llama2-Onnx\Llama-2-Onnx>

[mszhanyi]

I came acorss the same issue.
But it works if you use 7B_FT_Float16
python MinimumExample/Example_ONNX_LlamaV2.py --onnx_file 7B_FT_float16/ONNX/LlamaV2_7B_FT_float16.onnx --embedding_file 7B_FT_float16/embeddings.pth --tokenizer_path tokenizer.model --prompt "What is the lightest element?"