[Spec Decode][CI] Add e2e test for examples/spec_decode.py and prevent breaking Acceptance Length

[ekagra-ranjan]

In the past, the examples/offline_inference/spec_decode.py has broken a few times due to changes in datasets or other places like this. This script is an important one since this allows measuring AL for SD methods.

This PR adds this script to CI and ensures 2 things

1. the example script is in working condition
2. the AL of default method, i.e., Eagle, is measured during CI as an e2e test since many SD code path use Eagle related components.

Testing

cmd
time python3 examples/offline_inference/spec_decode.py --test --method eagle --num_spec_tokens 3 --dataset-name hf --dataset-path philschmid/mt-bench --num-prompts 80 --temp 0 --top-p 1.0 --top-k -1 --tp 1 --enable-chunked-prefill

Output

Adding requests: 100%|███████████████████████████████████████████████████████████████████████████████████| 80/80 [00:00<00:00, 12256.43it/s]
Processed prompts: 100%|██████████████████████████| 80/80 [00:02<00:00, 38.61it/s, est. speed input: 3886.77 toks/s, output: 8174.11 toks/s]
--------------------------------------------------
total_num_output_tokens: 16936
num_drafts: 7403
num_draft_tokens: 22209
num_accepted_tokens: 9535
mean acceptance length: 2.29
--------------------------------------------------
acceptance at token 0: 0.68
acceptance at token 1: 0.39
acceptance at token 2: 0.21
Test passed!

real    0m31.999s
user    0m52.270s
sys     0m6.709s

[gemini-code-assist]

Code Review

This pull request introduces an end-to-end test for the spec_decode.py example script, integrating it into the CI pipeline to safeguard against regressions in speculative decoding acceptance length. The approach of refactoring the script for testability and adding assertions for a fixed test case is sound. My review focuses on improving the robustness and maintainability of these new tests. I've identified a missing assertion for a critical test parameter and a formatting issue in an assertion message that could hinder debugging. Addressing these points will make the new test more reliable.

[wwl2755]

Great job for maintaining this! I think it is worthwhile to maintain the example scripts valid through CI, since they may be the places people get started from.

Link to #22992 for visibility.

[benchislett]

It's not clear to me if this should be EAGLE1 or EAGLE3 or both, but in any case this is good to have.

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @ekagra-ranjan.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork