LLM.int8() Refactoring: Part 1

[matthewdouglas]

This PR is the initial phase of a set of changes aimed at improving the LLM.int8() implementation.

@TimDettmers @Titus-von-Koeller

Primary Purpose

• Introduces support for Hopper (H100, H200) GPUs.
• Resolves [Bug] Exception: cublasLt ran into an error! during fine-tuning LLM in 8bit mode #538

Enhancements

• Removes the usage of Turing and Ampere specific memory layouts while retaining compatibility across sm_75 through sm_89.
  • Simplifies the code and surface area needing to be maintained.
  • Reduced overhead by removing layout transformation operations.
• Removes the separate NO_CUBLASLT build while retaining compatibility for targets below sm_75. ^{verification in progress}
  • This simplifies building and packaging, and trims the size of binary wheels in ~half.
• Support for CUDA Graph tracing to bring parity with 4bit.
• Improved kernels for inference:
  • Fused kernel for activation scale calibration and quantization. (Exposed as op F.int8_vectorwise_quant)
  • Other kernels simplified to operate with row-major data.
• Makes many unit tests more reliable with increased determinism.

Deprecations

The following functions from bitsandbytes are deprecated:

mm_cublas
bmm_cublas
matmul_cublas

The following functions from bitsandbytes.functional are deprecated:

_mul
arange
dequant_min_max
dequantize_no_absmax
extract_outliers
get_special_format_str
get_tensor_stream (moved to internal API)
get_transform_buffer
get_transform_func
mm_dequant (replacement: int8_mm_dequant)
igemmlt (replacement: int8_linear_matmul)
nvidia_transform
post_call
pre_call
transform
quantize_no_absmax
vectorwise_dequant
vectorwise_quant (~replacement: int8_vectorwise_quant)
vectorwise_mm_dequant (~replacement: int8_mm_dequant)

Further testing and benchmarking will be coming. At the moment unit tests are passing.

Next steps

• Clean up and reorganize unit tests
• Documentation for public APIs
• Ensure fallback path for shapes that don't work well with cuBLASLt (i.e. m/k not multiples of 4).
• Add an int8 dequantize op
• Further improvement of sparse decomposition performance (Deferred to future PRs)
• Conduct profiling, benchmarks, and evaluations
  • Build benchmark/evaluation scripts
  • Prepare analysis of results

[github-actions]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[matthewdouglas]

We support Python 3.8+ only, so use the builtin.

[matthewdouglas]

This won't be used anymore but I'm not sure of the side-effects of removing these properties either. Could be downstream integrations accessing them, maybe used in serialization etc. Any tips/thoughts here are welcome.

[matthewdouglas]

We skip this for inference now as it's also not needed.

[matthewdouglas]

This isn't specific for int8, but while I was profiling I noticed an opportunity to slightly improve some of the overhead here.

[matthewdouglas]

Not int8 but another small 4bit change that wanted to sneak its way in. @TimDettmers I'm just looking for a sanity check here that this makes sense.

[matthewdouglas]

There have now been some documentation updates both for the inline docstrings and the markdown-format public docs.

Additionally, tests related to 8bit now use static shapes. Certain tests related to benchmarking have been extracted away, and others have had a new deprecated marker applied where appropriate.

A more detailed look at benchmarking data will be provided with release materials. For now, an overview of inference benchmark results:

• INT8
  • On T4 and 4090, the per-token throughput is improved by 60-85% and per-token latency is decreased by 40-45%.
  • H100 is now supported. With Llama 3.1 70B and batch size >= 8, INT8 is consistently faster than NF4.
• NF4:
  • On T4 and 4090, with batch size of 1, per-token throughput is improved by 10-25% and per-token latency is decreased by 10-20%.
  • On H100, across all batch sizes, per-token throughput is improved by up to 28% and per-token latency is decreased by up to 22%.

[Titus-von-Koeller]

Really well done @matthewdouglas! What a thorough and well done refactor, which definitely needed some serious skill and dedication: Excellent work and results!

Imo this is ready to merge now, just the docs deep dive and some small improvements that we discussed in Slack and then 🚀✨

[matthewdouglas]

Benchmark details have been added. I've also confirmed that everything is working on V100 without the separate NO_CUBLASLT build needed.