Merged ONNX decoder next steps

[fxmarty]

Feature request

The PR #647 was merged that adds support for merged without/with past decoder as a single ONNX file, along with inference in ORTModelForCausalLM.

Some key steps are still remaining:

• Support IO Binding Support IO Binding for merged decoder #797
• Find a way to hide the prints 2023-02-10 16:29:24.868007832 [W:onnxruntime:, graph.cc:3487 CleanUnusedInitializersAndNodeArgs] Removing initializer '/transformer/h.4/attn/Constant_18_output_0'. It is not used by any node and should be removed from the model., tracked in CleanUnusedInitializersAndNodeArgs warnings are printed only with subgraphs microsoft/onnxruntime#14694
• Fix the generation of dummy past key values for bloom that is currently ugly
• Investigate why codegen does not support -with-past in tasks.py
• Support the merge for Seq2Seq model
• Support ONNX Runtime optimizations along with merged models Support ONNX Runtime optimizations in exporters.onnx #807

Motivation

Reduce memory usage

Your contribution

/

[fxmarty]

Hi @un-certainty , yes if you are using CUDAExecutionProvider, using IO Binding is probably helpful. I don't have a proper benchmark at hand though.

Also I wonder if the caches are perserved on GPU, will it potentially cause a memory explosion? When the QPS is high and sequences are long, there will be a lot of intermediate tensors. I'm not sure if this could lead to OOM.

I would say it could, yes.