[FEATURE]: more comprehensive e2e testings in per-merge CIs with mockers

[PeaBrane]

Feature request

The minimum functionality testing of our Router with mockers is introduced in this PR #2073 . This can be made more comprehensive, for example checking the validity of Indexer, load metrics, and various overheads.

Describe the problem you're encountering

N/A

Describe alternatives you've tried

No response

[zhixiongli2011]

@PeaBrane Is there anything I can help here? Is this the right understanding of this ticket: create more testings like the below PR #2324 to comprehensively test the Router to identify code defects, performance issues, etc. or just for the automation/regression test? I would like to give it a try. Please advise.

Or please suggest any other tickets that I can work on as the new comer with your directions?

[PeaBrane]

Yes! So one idea is to augment our existing e2e python binded router test (the one with mockers). You can send some random list of integers to a backend worker you specify (as worker id kwarg). Wait a bit. Then a second list of integers with some prefix overlap, verify that the best worker id and overlap score is as expected. Or anything along this line.

[zhixiongli2011]

@PeaBrane That is a good idea. Thanks for the information. I will try and let you know.

[zhixiongli2011]

@PeaBrane Good morning, could you please help verify one thing: if the workload (GPU usage) of the workers is currently used to select the best worker together with the KV-cache? The below screenshot was from the Dynamo demonstration at the GTC in March. However, the source code doesn't have anything about the GPU metric and uses overlap_weight + prefill block + decode block + temperature to make the selection decision. Did I check the right code? I am trying to understand the routing logics so to create the good test cases with the right expected results.

44:40-48:00 https://youtu.be/3C-6STonTLU?si=-fffsKfvfhpKyqaR

https://github.com/ai-dynamo/dynamo/blob/main/lib/llm/src/kv_router/scheduler.rs

[PeaBrane]

yea, the GTC info is old. We simplified the cost function, and don't actaully listen to load metrics from the engines anymore. But we still listen to the KV events

[zhixiongli2011]

Thanks for the confirmation.

[zhixiongli2011]

@PeaBrane Hope you have had a good day today. I have eventually run test (test_router_e2e_with_mockers.py) successfully on my VM after fixed a few issues of the environment configuration and about to implement the below test case. Please help verify if it is a valid test case.

Test case 1 (base case):

1. create two mocker instances and get the instance id of each
2. initialize and check the readiness of both instances by sending dummy_token_ids = [1, 2, 3] to each instence using the instance id
3. send test_token_ids_1 with 100 random.randint(1, 10000) to mocker instance 1
4. generate a new test_token_ids_2 = test_token_ids_1 + additional 10 random.randint(1, 10000)
5. call KvPushRouter.generate using the "query_instance_id" annotation and test_token_ids_2
6. expected result: id of mocker instance 1 should be returned as the worker_instance_id regardless of the value of overlap_weight.

Meanwhile I will do more test to understand what "decode_block" is in the below statement. The comment says, "this is the number of decode blocks the worker would have if the request were scheduled there". is it the total number of blocks that will be generated for the current query (input tokens) or decode blocks that have been generated by all previous queries or something else? The former seems not right because it makes the overlap_weight useless.

let logit = overlap_weight * potential_prefill_block + decode_block

[PeaBrane]

Yes this sounds like a good test.

Decode block is the total number of KV blocks that is being actively used by the engine (or equivalently blocks that cannot be evicted and must stay on vram), plus the new potential blocks that the new request would add

[zhixiongli2011]

Thanks for the confirmation and the additional information.