Add ORCA endpoint load metrics support

[efimki]

Purpose

Add ORCA endpoint load metrics for /chat/completion and /completion responses.

• Engine V1 implementation uses get_named_metrics_from_prometheus() to collect metrics.
• Reporting of engine metrics in response headers when enabled through request header "endpoint-load-metrics-format" supported formats "text" and "json".
• No added load on response for default case
• No new computation in engine

Test Plan

Added tests/entrypoints/openai/test_orca_metrics.py
Run: pytest -s -v tests/entrypoints/openai/test_orca_metrics.py

Test Result

Pass

FIX

#10086

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @efimki.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[mergify]

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @efimki.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

[robertgshaw2-redhat]

@markmc

[efimki]

@markmc - do you have any comments / suggestions?

[markmc]

If you rebase, you'll see v0 engine is removed by #25033 and I think that means you can remove the InbandEngineStats stuff

Also, can 'get_metrics_snapshot()' do what you want? Maybe we can add a filter to that API so you can just get the two metrics you want?

[markmc]

For reference, (even for myself), this background info is useful as justification for adding these in-band metrics: #14906 (comment)

[efimki]

If you rebase, you'll see v0 engine is removed by #25033 and I think that means you can remove the InbandEngineStats stuff

Excellent point, it made this PR much simpler!

Also, can 'get_metrics_snapshot()' do what you want? Maybe we can add a filter to that API so you can just get the two metrics you want?

The get_metrics_snapshot() API looks good. Adding a filter could be useful to avoid overhead of converting all the metrics. At this point I'm not entirely sure what metrics would be used for load balancing, but we can tweak it later.

[efimki]

Friendly ping. :)

Any comments and suggestions would be greatly appreciated. :)

[efimki]

@markmc - thanks for the approval! Do you know who else needs to approve it for merging?

[simon-mo]

@markmc reviewed

[markmc]

Needs a rebase to fix conflicts and pick up #27610 which fixes the buildkite/ci/pr/entrypoints-integration-test-api-server failure

[efimki]

Needs a rebase to fix conflicts and pick up #27610 which fixes the buildkite/ci/pr/entrypoints-integration-test-api-server failure

Thanks for the suggestion! I'm having some issues with my environment missing required version of torch:

  × No solution found when resolving dependencies:
  ╰─▶ Because there is no version of torch==2.9.0+cu129 and you require torch==2.9.0+cu129, we can conclude that your requirements are unsatisfiable. 

I'll try to fix that ASAP.

[markmc]

Another rebase/merge needed to pick up #27861 and #27811