Generating output from finetuned model using LLM.generate() method vs litgpt chat cli command

[asmitnayak]

I performed lora finetuning on Qwen2.5-3B-Instruct model on my custom dataset, and it seems that when I generate content using the litgpt chat cli interface the finetuned model performs flawlessly. However, when using it through the python api, following the tutorial the model outputs are random characters and basically rubbish. I tried using litserve and converting it into HuggingFace models but none worked.

Does anyone have any idea how to fix this.

P.S. Currently I am using a custom implementation of litgpt\chat\base.py to get my code working but this is pretty slow and I wish to parallelize the generation.

Thanks for any help.