Quesiton on the speed for generating the response #111
Comments
|
Hi! What gpu did you use ? You need at least Ampere |
|
hi im using 4090 |
|
Can you try this, and run with Also, did you install BitBlas before ?
import torch
from transformers import AutoTokenizer
from hqq.models.hf.base import AutoHQQHFModel
from hqq.utils.patching import *
from hqq.core.quantize import *
from hqq.utils.generation_hf import HFGenerator
#Load the model
###################################################
#model_id = 'mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq' #no calib version
model_id = 'mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib' #calibrated version
compute_dtype = torch.bfloat16 #bfloat16 for torchao, float16 for bitblas
cache_dir = '.'
device = 'cuda:0`
model = AutoHQQHFModel.from_quantized(model_id, cache_dir=cache_dir, compute_dtype=compute_dtype, device=device)
tokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir=cache_dir)
quant_config = BaseQuantizeConfig(nbits=4, group_size=64, quant_scale=False, quant_zero=False, axis=1)
patch_linearlayers(model, patch_add_quant_config, quant_config)
#Use optimized inference kernels
###################################################
HQQLinear.set_backend(HQQBackend.PYTORCH)
#prepare_for_inference(model) #default backend
prepare_for_inference(model, backend="torchao_int4")
#prepare_for_inference(model, backend="bitblas") #takes a while to init...
#Generate
###################################################
#For longer context, make sure to allocate enough cache via the cache_size= parameter
gen = HFGenerator(model, tokenizer, max_new_tokens=1000, do_sample=True, compile="partial").warmup() #Warm-up takes a while
import time
t1 = time.time()
out = gen.generate("Write an essay about large language models", print_tokens=False)
t2 = time.time()
print('Took', t2-t1, 'secs') |
|
I am getting ~138 tokens per sec, which is about 6-7 seconds end-to-end, depending on the output (since sampling is enabled |
|
yeah i did install and received the warning |
|
I tried running the code you gave me, and it showed this error |
|
It seems like you are using an older version of Pytorch, can you try the following: Step 1: InstallationMake sure you use the right CUDA version (here it's using 12.1) Step 2: Runthen you can run the code: import torch
from transformers import AutoTokenizer
from hqq.models.hf.base import AutoHQQHFModel
from hqq.utils.patching import *
from hqq.core.quantize import *
from hqq.utils.generation_hf import HFGenerator
#Load the model
###################################################
#model_id = 'mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq' #no calib version
model_id = 'mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib' #calibrated version
compute_dtype = torch.bfloat16 #bfloat16 for torchao, float16 for bitblas
cache_dir = '.'
device = 'cuda:0`
model = AutoHQQHFModel.from_quantized(model_id, cache_dir=cache_dir, compute_dtype=compute_dtype, device=device)
tokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir=cache_dir)
quant_config = BaseQuantizeConfig(nbits=4, group_size=64, quant_scale=False, quant_zero=False, axis=1)
patch_linearlayers(model, patch_add_quant_config, quant_config)
#Use optimized inference kernels
###################################################
HQQLinear.set_backend(HQQBackend.PYTORCH)
#prepare_for_inference(model) #default backend
prepare_for_inference(model, backend="torchao_int4")
#prepare_for_inference(model, backend="bitblas") #takes a while to init...
#Generate
###################################################
#For longer context, make sure to allocate enough cache via the cache_size= parameter
gen = HFGenerator(model, tokenizer, max_new_tokens=1000, do_sample=True, compile="partial").warmup() #Warm-up takes a while
import time
t1 = time.time()
out = gen.generate("Write an essay about large language models", print_tokens=False)
t2 = time.time()
print('Took', t2-t1, 'secs') |
|
i tried following all the step and it shows this error |
|
this is the full error The above exception was the direct cause of the following exception: Traceback (most recent call last): Set TORCH_LOGS="+dynamo" and TORCHDYNAMO_VERBOSE=1 for more information You can suppress this exception and fall back to eager by setting: |
|
The error message says that you don't have Also install triton if it was not installed |
|
torch version :2.5.0.dev20240825+cu121 |
|
Oh you are right, seems like it's not supported on Windows: triton-lang/triton#1640 😞 |
|
owh understand! thanks for the reply and explanation will try on the linux later ! 👍 |
|
Sorry, one last question. Can I know what the format is after quantization using HQQ? Is it also in .pt, safetensor, or gguf? |
|
The format is custom only compatible with transformers. I am currently working to add full support directly on transformers huggingface/transformers#33141 |
|
Oh, okay, understand. Can you share some thoughts or guidance on how to make the HQQ quantized model work in VLLM? |
|
I don't use it personally, but this branch implements the fast backends used in HQQ, but I can't find a clean example to see how it works |
|
understand thanks for the reply! 😃 |
Can I ask, I tried following the usage on Hugging Face, but it takes almost 100 seconds to generate a response to the question 'Write an essay about large language models.' However, when I use the ollama/llama3.1 model directly, it only takes about 7-8 seconds to generate the response. Is this normal?
This is the code i using
import torch
from transformers import AutoTokenizer
from hqq.models.hf.base import AutoHQQHFModel
from hqq.utils.patching import *
from hqq.core.quantize import *
from hqq.utils.generation_hf import HFGenerator
#Load the model
###################################################
#model_id = 'mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq' #no calib version
model_id = 'mobiuslabsgmbh/Llama-3.1-8b-instruct_4bitgs64_hqq_calib' #calibrated version
compute_dtype = torch.float16 #bfloat16 for torchao, float16 for bitblas
cache_dir = '.'
model = AutoHQQHFModel.from_quantized(model_id, cache_dir=cache_dir, compute_dtype=compute_dtype, map_location=torch.device('cuda'))
tokenizer = AutoTokenizer.from_pretrained(model_id, cache_dir=cache_dir)
quant_config = BaseQuantizeConfig(nbits=4, group_size=64, quant_scale=False, quant_zero=False, axis=1)
patch_linearlayers(model, patch_add_quant_config, quant_config)
#Use optimized inference kernels
###################################################
HQQLinear.set_backend(HQQBackend.PYTORCH)
#prepare_for_inference(model) #default backend
#prepare_for_inference(model, backend="torchao_int4")
prepare_for_inference(model, backend="bitblas") #takes a while to init...
#Generate
###################################################
#For longer context, make sure to allocate enough cache via the cache_size= parameter
gen = HFGenerator(model, tokenizer, max_new_tokens=1000, do_sample=True, compile="partial").warmup() #Warm-up takes a while
gen.generate("Write an essay about large language models", print_tokens=True)
gen.generate("Tell me a funny joke!", print_tokens=True)
gen.generate("How to make a yummy chocolate cake?", print_tokens=True)
The text was updated successfully, but these errors were encountered: