What is the right data for SOURCE_LORA when Convert LoRA to cpp format

[sleepwalker2017]

I'm benchmarking lora following this: https://github.com/NVIDIA/TensorRT-LLM/blob/main/benchmarks/cpp/README.md#benchmarking-lora

I clone this repo: https://huggingface.co/hfl/chinese-llama-2-lora-13b

when I run the script:

# Convert LoRA to cpp format
python examples/gpt/nemo_lora_convert.py \
    -i $SOURCE_LORA \
    --storage-type $DTYPE \
    --write-cpp-runtime-tensors \
    -o $CPP_LORA

It reports error.

    with tempfile.TemporaryDirectory() as prompt_out_dir:
        prompt_out_dir = Path(prompt_out_dir)
        unpack_nemo_ckpt(args.in_file, prompt_out_dir)
        LOGGER.info("Spent %s (h:m:s) to unpack NeMo prompt archive",
                    datetime.datetime.now() - start_time)

        model_weights_ckpt = "model_weights.ckpt"
        with open(prompt_out_dir / "model_config.yaml") as f:
            prompt_config = yaml.full_load(f)
        LOGGER.debug(prompt_config)

It looks like the file need to open a tar file, but this model is a folder.

And also, the lora weight folder contains no model_weights.ckpt and no model_config.yaml.
How can I run this script? Any advice ?

[byshiue]

The nemo converter is only used for nemo ckpt, you cannot use it on huggingface ckpt.

[sleepwalker2017]

The nemo converter is only used for nemo ckpt, you cannot use it on huggingface ckpt.

So what should we do at this step ? I want to benchmark trt-llm for multi lora, I get stuck here.

[byshiue]

This is a miss of the document. You should use examples/hf_lora_convert.py like

# Convert LoRA to cpp format
python examples/hf_lora_convert.py \
    -i $SOURCE_LORA \
    --storage-type $DTYPE \
    -o $CPP_LORA

Also, the NUM_LORA_MODS is 7 in this case. We will fix it soon.

[sleepwalker2017]

NUM_LORA_MODS

NUM_LORA_MODS what does this mean? q k v o proj? and up down gate proj?

[sleepwalker2017]

This is a miss of the document. You should use examples/hf_lora_convert.py like

# Convert LoRA to cpp format
python examples/hf_lora_convert.py \
    -i $SOURCE_LORA \
    --storage-type $DTYPE \
    -o $CPP_LORA

Also, the NUM_LORA_MODS is 7 in this case. We will fix it soon.

#1552
I get this issue when running lora, could you take a look at this?