feature: support for Kubernetes Gateway Inference Extensions

[AndresGuedez]

Describe the feature

See https://github.com/kubernetes-sigs/gateway-api-inference-extension.

These Inference Extensions are being formally specified as part of the Kubernetes Gateway API and will provide a standard interface for inference optimized load balancing algorithms such as the production stack Router.

Why do you need this feature?

Provides a (soon-to-be) standardized integration with Kubernetes Gateways.

Additional context

No response

[AndresGuedez]

cc: @rootfs @KuntaiDu @ApostaC

[rootfs]

• From the API standards, there are a few concepts (i.e. inference pool) that are not immediately applicable here but can be translated from the e.g. service discovery mechanisms.
• The Envoy extproc integration is currently experimented in [FEAT] Integrate the router with Envoy through extproc #240, but apparently needs some alignment.

[Xunzhuo]

Hi @rootfs Can tell the current state of traffic before adding eg support? And then a little bit goals about the scenario of using eg?

This is a big enhancement, we need clear goals and roadmap for this : )

[rootfs]

@Xunzhuo the current implementation in #240 serves the purpose of how to integrate the router package with a new frontend (i.e. extproc) and enables certain configurations (i.e. semantic cache). The work is still in progress, it would be great if you and other other GW API extension team can review it and come up with yet another design. Please feel free to ping me in vLLM slack production-stack channel.

[rootfs]

@AndresGuedez @Xunzhuo would you be good to collaborate on this on a feature branch (if @YuhanLiu11 @ApostaC can create one for us)?

[rootfs]

@AndresGuedez @YuhanLiu11 @ApostaC @Xunzhuo I did some research on the GWAPI EX and other projects. I believe the existing APIs don't offer the full cycle management of inference services. I am creating a proposal here, and wanting to use this as a place for discussion and potential PoC for advancing the API scope and capabilities.

[ahg-g]

@rootfs can you elaborate on what you are including in "full cycle management"? The inference gateway and its APIs aim to define extension points with k8s load balancers to allow:

1. Executing inference-optimized load balancing algorithms
2. Dense serving of LoRA adapters
3. Controlled rollouts of adapters
4. Model-aware priority and fairness
5. Emitting inference metrics for both observability and autoscaling

A key point here is to offer that in a way that composes with users' existing workflows and toolings with inference gateway, a user that already uses k8s Gateway API can easily get all the above benefits.

[kfswain]

I read through: #319 and some of these lower level CRDs read like they are breaking up any vLLM into layers?

the documentation is pretty sparse so I had to infer a decent amount, I could be misinterpreting

[rootfs]

@ahg-g @kfswain @YuhanLiu11 thanks for the review. I do have a detailed google doc in the works, will share it after review. I think we should meet to discuss it, would you be available this Thursday or Friday? I am in US ET.

[YuhanLiu11]

@ahg-g @kfswain @YuhanLiu11 thanks for the review. I do have a detailed google doc in the works, will share it after review. I think we should meet to discuss it, would you be available this Thursday or Friday? I am in US ET.

Meeting on Thursday or Friday works for me! I'm generally free on these two days. Feel free to propose time slots that work best for you guys!

Thanks.

[ahg-g]

I am available Friday if that works for everyone. It would be great if you can share the doc beforehand.

[rootfs]

I am available Friday if that works for everyone. It would be great if you can share the doc beforehand.

@YuhanLiu11 @ahg-g would 2PM ET/11AM PT work for all?
@ahg-g yes, will send the doc soon.

[YuhanLiu11]

I am available Friday if that works for everyone. It would be great if you can share the doc beforehand.

@YuhanLiu11 @ahg-g would 2PM ET/11AM PT work for all? @ahg-g yes, will send the doc soon.

That works for me! @ahg-g @kfswain if that works for you too, I'll send a calendar invite soon! Thanks.

[kfswain]

Yup, Friday sounds great