Are benchmarks fair?

[SaeedNajafi]

It seems that your codebase has a separate implementation for Multi-head attention (MHA module) along with a separate implementation for kv caching and even the generation function is different than HF's generation.

While you are loading HF's models, you are relying on HF implementations. Could this introduce discrepancies in benchmarks?
Is it possible to build a transformer model using only your codebase relying on the local implementation of kv cache and MHA implementations?

mamba/benchmarks/benchmark_generation_mamba_simple.py

Line 41 in 442fab4

model = AutoModelForCausalLM.from_pretrained(args.model_name, device_map={"": device}, torch_dtype=dtype)