Update TensorRT-LLM

[kaiyux]

• Model Support
  • BART and mBART support in encoder-decoder models
  • Support FairSeq Neural Machine Translation (NMT) family
  • Mixtral-8x7B model support
    • Support weight loading for HuggingFace Mixtral model
• Features
  • MPT - Int4 AWQ / SmoothQuant support
  • Support speculative decoding with prefilled KV cache
  • Support AWQ and GPTQ support for QWEN
  • Support ReduceScatter plugin
• Bug fixes
  • Fix tokenizer usage in quantize.py Llama 2 FP8 quantization OOM #288, thanks to the contribution from @0xymoro
  • Fix LLaMa with LoRA error [Llama Model] Run LLaMa with LoRA Error #637
  • Fix the first token being abnormal issue when '--gather_all_token_logits' is enabled When '--gather_all_token_logits' is enabled, the first token appears to be abnormal." #639
  • Fix Expert and Tensor Parallelism for Mixture of Expert layers
  • Fix an accuracy bug in MMHA multi_block_mode when certain conditions are met (large TP & 32K sequence length)
  • Fix BERT FMHA flag
• Performance
  • Optimize Hopper warp specialized kernels
  • Optimize AllReduce for parallel attention on Falcon and GPT-J
  • Enable split-k for weight-only cutlass kernel when SM>=75

[leavelet]

There seems to be a mistake here? In a multi-node setup, the product of mGpusPerNode and the number of nodes should exceed the total of TP multiplied by PP. However, in this case, the number of GPUs per node is actually less than the product of TP and PP
@kaiyux

[juney-nvidia]

Thanks for the sharp catch. I will discuss with the TensorRT-LLM engineers working on C++ runtime and go back to you later.

June

[leavelet]

Hello, is there any progress on this?

[kaiyux]

Hi @leavelet , we did not fully test multi-node support in TensorRT-LLM, it might be working, but there is no guarantee. If you need to try that, we suggest that you remove the check locally so that it won't block you. Thanks very much for your interest on our work!