[Model] Add T5 model (2/2)

[NickLucche]

Add support fot T5 (encoder-decoder model).
Follow-up and based on #11334, so it needs this other PR merged before it can be addressed, as it assumes to have a backend that supports passing a custom attention bias in both prefill and decode (xformers+pagedattention as of now).

Some topics I'd like to discuss here (RFC):

• xFormers is a hard dependency for T5 as of now, do we silently enforce the backend or just raise an error? Surely we need to raise one on platforms xformers does not support (rocm)
• T5 uses the same decoder_start_token_id as it does for padding, but there's no explicit BOS. Current logic in preprocess.py would just crash. Is this the best approach to handle the quirk of T5?
• changes to xformers.py. I'd rather have spared this one, but it was assuming alibi_slopes were the only way to have multiple attention biases (one per sequence).

Additional items:

• Cuda Graph support

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.
Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can do one of these:

• Add ready label to the PR
• Enable auto-merge.

🚀

[NickLucche]

P.S I've tested in eager mode only so far, still have to investigate whether cuda graphing this one may cause issues or not

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @NickLucche.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[vhminh2210]

@NickLucche Thanks for implementing this PR!

On testing your PR, I encountered some unexpected behaviors, and hope to get your insights on the problems

Describe the issues:

1. When sending requests using python multiprocess to simulate batch inferences, the vLLM output text of the same queries are different between runs and different from Huggingface (HF) output:

[MODEL] : google-t5/t5-small
[PROMPT]: "After Roger 's death , his son inherited -- William de Roumare , Earl of Lincoln -- the manor house ."
[vLLM]  : "Roger 's death , his son inherited -- William de Roumare , Earl" (SINGLE-PROCESS)
        : "Roger 's son inherited the manor house after Roger's death ." (MULTI-PROCESS-RUN-1)
        : "Roger 's son inherited the manor house ." (MULTI-PROCESS-RUN-2)
        : "Roger 's death , his son inherited -- William de Roumare , Earl" (MULTI-PROCESS-RUN-3)
[HF]    : "Roger's death, his son inherited -- William de Roumare, Earl"

In general, the output quality decreases when inferring by batches.

2. There are chances that T5-based model output text with only <pad>'s when serving with vLLM (eager mode) while having different output using HF. The behaviour become more frequent with large models. For example, t5-large outputs only full-<pad> texts over 500 test queries.

[MODE]  : SINGLE-PROCESS
[MODEL] : ul2-tiny-nl6-finnish
[PROMPT]: "The body is white , limbs and fingers are black and the tail is long ."
[vLLM]  : "<pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>"
[HF]    : "_ on nyt. . . . . . . . . . . . . . . . ."

[MODE]  : SINGLE-PROCESS
[MODEL] : google-t5/t5-base
[PROMPT]: "In this way , under Nestorian auspices , the tragic faith was established in the East ."
[vLLM]  : "<pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad><pad>"
[HF]    : "<extra_id_0> , the tragic faith was established in the East ."

3. T5-based models have different outputs when inferred using vLLM and Huggingface (HF), even on SINGLE-PROCESS testing. For example:

[MODE]  : SINGLE-PROCESS
[MODEL] : google-t5/t5-small
[PROMPT]: "After Roger 's death , his son inherited -- William de Roumare , Earl of Lincoln -- the manor house ."
[vLLM]  : "Roger 's death , his son inherited -- William de Roumare , Earl"
[HF]    : "Roger's death, his son inherited -- William de Roumare, Earl"

[MODE]  : SINGLE-PROCESS
[MODEL] : google-t5/t5-small
[PROMPT]: "Charley Frazier ( born August 12 , 1939 in Houston , Texas ) is a former American football receiver in the American Football League and the NFL ."
[vLLM]  : "Charley Frazier (* 12. August 1939 in Houston , Texas ; <unk> 12."
[HF]    : "Charley Frazier (* 12. August 1939 in Houston, Texas ; <unk>12."

Steps to reproduce:

1. Install vLLM from source by full build from the PR version with compilation following https://docs.vllm.ai/en/stable/getting_started/installation/gpu/index.html?device=cuda#build-wheel-from-source

git clone --branch t5-model --single-branch https://github.com/NickLucche/vllm.git
cd vllm
pip install .

2. Serve model (for e.g., t5-small) with vLLM on eager mode:

vllm serve --enforce-eager google-t5/t5-small

3. Remove the assert at line 355 of https://github.com/NickLucche/vllm/blob/t5-model/vllm/model_executor/models/t5.py#L355 to avoid attn_bias not None error.

assert attn_bias is None  # to be recomputed

4. Query the model by sending requests in the form:

curl http://localhost:8000/v1/completions \
    -H "Content-Type: application/json" \
    -d '{
        "model": "google-t5/t5-base",
        "prompt": "Describe San Francisco",
        "max_tokens": 20,
        "skip_special_tokens": false,
        "seed": 42,
        "temperature": 0,
        "top_k": 1,
        "top_p": 1.0
    }'

The requests are sent using Python multiprocess, with processes=1 for single-process testing, and set default for multi-processes testing, given that the machine nproc=16 and the testset has 500 queries.

5. The corresponding parameters are set for HF benchmark outputs:

tokenize_results = tokenizer(prompt, return_tensors="pt", padding= True, return_attention_mask= True)
input_ids, attn_mask = tokenize_results["input_ids"], tokenize_results["attention_mask"]
outputs = model.generate(input_ids= input_ids, attention_mask= attn_mask, do_sample= False, temperature= 0, top_k= 0, top_p= 0.0)

6. The model used for testing:

• google-t5/t5-small, t5-base and t5-large
• Finnish-NLP/ul2-tiny-nl6-finnish, ul2-small-nl24-finnish and ul2-base-nl36-finnish

The full-<pad> phenomenon can be observed with google-t5/t5-large, Finnish-NLP/ul2-small-nl24-finnish and Finnish-NLP/ul2-base-nl36-finnish

7. The output HF texts are post-processed by removing beginning <pad> and ending </s> with follow-up tokens. All output texts (generated by HF or vLLM) are .strip() to remove leading \s, \n, ...

@NickLucche, do you have any insights into what might be causing the problems?

[NickLucche]

Hey thanks a lot for giving it a spin! Are you able to observe the same behavior with google/flan-t5-base?
I will take a look at this old PR asap, though I will first have to rebase part 1.

Further testing with all params you've tried is surely needed, but the assert at https://github.com/NickLucche/vllm/blob/t5-model/vllm/model_executor/models/t5.py#L355 should not be removed. The fact that it raises on that condition is worth looking into on its own.

[vhminh2210]

Hey thanks a lot for giving it a spin! Are you able to observe the same behavior with google/flan-t5-base? I will take a look at this old PR asap, though I will first have to rebase part 1.

Further testing with all params you've tried is surely needed, but the assert at https://github.com/NickLucche/vllm/blob/t5-model/vllm/model_executor/models/t5.py#L355 should not be removed. The fact that it raises on that condition is worth looking into on its own.

I did a quick check by enabling the assertion in --enforce-eager mode and everything seems fine so far. However, without --enforce-eager, the command vllm serve Finnish-NLP/ul2-base-nl36-finnish failed with an AssertionError, which stems from assert attn_bias is None. I think this behavior is notable and hope this can give you some insights into the problem.

[NickLucche]

Thanks, that would match with the setup we currently have in tests.

[NickLucche]

Hey @vhminh2210 I could use your help to turn some of your experiments into unit tests to add. What models would you want to see tested? I am possibly looking for fine tuned T5 models, the base models such as Finnish-NLP/ul2-small-nl24-finnish end up having collapsed outputs with current prompts, being harder to verify.