Update on the development branch

[kaiyux]

Hi,

The TensorRT-LLM team is pleased to announce that we are pushing an update to the development branch(also includes the Triton backend) this November 17th, 2023.

This update includes:

• Model Support
  • QWEN (external contribution - thank to @Tlntin and @zhaohb)
  • Replit Code V-1.5 3B (external contribution Add GQA support to MPT (and GPT) models #205 - thanks to @bheilbrun)
• Features
  • MPT MQA/GQA support (external contribution Add GQA support to MPT (and GPT) models #205 - thanks to @bheilbrun)
  • Falcon INT4 AWQ support
  • Encoder-decoder models
    • BF16 support
    • Multi-GPU support, including tensor parallelism, pipeline parallelism and hybrid parallelism.
    • T5 family support w/ gated activation, including T5/mT5/Flan-T5 (developed in parallel with Add support for Flan-T5 models #351 - thanks to @mlmonk and @nqbao11)
    • Remove padding support
  • Limited support for LoRA (limited to QKV projection and only exposed in Python runtime for now) in the GPT model. More support will be released in the coming weeks.
  • C++ runtime
    • Support to gather generation-phase logits.
    • Add options to return logProbs and cumLogProbs
  • CUTLASS 3.x upgrade
  • Experimental Python runtime API
• Bug fixes
  • Tensor names error RuntimeError: Tensor names (host_max_kv_cache_length) in engine are not the same as expected on main branch #369
  • Bloom weights split issue world_size = 2 will raise error "array split does not result in an equal division" #374
  • SmoothQuant multi-GPU failure smooth-quant with tp=2, and build llama 7b with tp=2, pp=2 failed #267
  • GenerationSession crash issue GenerationSession crashes if stream keyword argument is not None #202
  • Typo when calling PyNVML API [BUG] code bug #410
• Memory optimization
  • Add utility to load model weight by shard to reduce peak CPU memory usage.
• Performance
  • Add reproduction steps for benchmarked performance results.
• Infra
  • NGC docker version upgraded from 23.08 to 23.10.
• Documentation
  • Add memory usage documentation.
  • Add instructions to run LLaMA that use Position Interpolation for long sequence lengths(32K)

Thanks,
The TensorRT-LLM Engineering Team

[mlmonk]

Thank you for these awesome updates. Does this mean that the enc_dec models now support inflight batching?

[shannonphu]

@symphonylyh Do you have any idea when Triton backend or inflight batching for enc_dec models can be available?

[symphonylyh]

@shannonphu enc-dec's Triton backend will come first likely in this month (i.e., next release). enc-dec's inflight batching is pending discussion and I cannot provide an estimate yet.

[shannonphu]

@symphonylyh is there any update on when enc-dec Triton backend will become available? I believe the latest releases/announcements did not mention it. Thanks in advance

[sihanwang41]

Hi @symphonylyh , we are also looking for enc_dec inflight batching support, it would be great to have this feature. Thanks!

[symphonylyh]

Hi @shannonphu @sihanwang41, yes we're working on it. Right now it's at the stage of adding the C++ runtime. Tentative date for Triton enc-dec support is around mid to late January. Thanks for your patience. btw, let's track the progress under #800