Int4(llama-2-chat-7b) converted model generates response in German language

[bhardwaj-nakul]

Describe the bug

1. Install all the pip dependencies for latest 254-llmchatbot notebook
2. Follow the steps to convert "llama-2-chat-7b" model to int4 format with default configuration.
3. Select device as "CPU"
4. Select model to run "INT4"
5. Run step to Load and compile the model
6. Set max_new_token=500 and run ov_model.generate with prompt "Describe Intel in 100 words or less"

Expected behavior

Output should be produced in English. However we are getting output in German language.

1. Is there any issue with converting llama-2-chat-7b model into int4 format with OpenVino ?
2. Is the issue with latest openvino==2023.3.0 or nncf==2.9.0.dev0+84b46f58 ?

Screenshots
Screenshot 1

Screenshot 2

Screenshot 3

Installation instructions (Please mark the checkbox)

• I followed the installation guide at https://github.com/openvinotoolkit/openvino_notebooks#-installation-guide to install the notebooks.

Additional context
I tried playing around with model_compression_params but it didn't help to resolve this issue.

"llama-2-chat-7b": {
    "mode": nncf.CompressWeightsMode.INT4_SYM,
    "group_size": 128,
    "ratio": 0.8,
},

"llama-2-chat-7b": {
    "mode": nncf.CompressWeightsMode.INT4_ASYM,
    "group_size": 128,
    "ratio": 0.8,
},

"llama-2-chat-7b": {
    "mode": nncf.CompressWeightsMode.INT4_SYM,
    "group_size": 64,
    "ratio": 0.8,
},

"llama-2-chat-7b": {
    "mode": nncf.CompressWeightsMode.INT4_SYM,
    "group_size": 64,
    "ratio": 0.6,
},

[openvino-dev-samples]

Hi @bhardwaj-nakul Since this model is for propose of chat, could you help to try adding a system prompt before the question, and see if the issue happen again?

[AlexKoff88]

@andreyanufr, can you please take a look and provide an advice on what kind of weight compression configuration should lead to accurate results?

[andreyanufr]

@bhardwaj-nakul could you please try to run data aware compression from attached file:
llama_example.txt

[andrei-kochin]

@bhardwaj-nakul could you please try the example suggested?

[bhardwaj-nakul]

I tried the provided script. I'm getting below error

The input model to this script is supposed to be FP16 IR model right ? I tried with this itself.

[AlexKoff88]

@bhardwaj-nakul, this is a known issue and it is already fixed. You should reinstall nncf from Github, like pip install git+https://github.com/openvinotoolkit/nncf.git