InferencePool "vllm-llama3-8b-instruct" not found

[Frapschen]

What happened:

Fellow guide docs: https://gateway-api-inference-extension.sigs.k8s.io/guides/#__tabbed_1_3, step by step, encouter bug with: kubectl get httproute llm-route -o yaml

root@controller-01:~/llm-d# kubectl get httproute llm-route -o yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1","kind":"HTTPRoute","metadata":{"annotations":{},"name":"llm-route","namespace":"default"},"spec":{"parentRefs":[{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"inference-gateway"}],"rules":[{"backendRefs":[{"group":"inference.networking.x-k8s.io","kind":"InferencePool","name":"vllm-llama3-8b-instruct"}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}],"timeouts":{"request":"300s"}}]}}
  creationTimestamp: "2025-09-04T06:14:44Z"
  generation: 1
  name: llm-route
  namespace: default
  resourceVersion: "60397"
  uid: 6b2bade6-ecf7-4dbf-9f1f-d2ba6c836cf1
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: vllm-llama3-8b-instruct
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
    timeouts:
      request: 300s
status:
  parents:
  - conditions:
    - lastTransitionTime: "2025-09-04T06:14:44Z"
      message: InferencePool "vllm-llama3-8b-instruct" not found
      observedGeneration: 1
      reason: BackendNotFound
      status: "False"
      type: ResolvedRefs
    - lastTransitionTime: "2025-09-04T06:14:44Z"
      message: ""
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    controllerName: kgateway.dev/kgateway
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: inference-gateway

here is my resource:

root@controller-01:~/llm-d# kubectl get pod
NAME                                          READY   STATUS    RESTARTS   AGE
inference-gateway-9c9675965-r5vsb             1/1     Running   0          18m
vllm-llama3-8b-instruct-6c9757687-4b5jt       1/1     Running   0          3h22m
vllm-llama3-8b-instruct-6c9757687-cw4h4       1/1     Running   0          3h22m
vllm-llama3-8b-instruct-6c9757687-f6zc2       1/1     Running   0          3h22m
vllm-llama3-8b-instruct-epp-7fc7b677f-b4sqj   1/1     Running   0          17m
root@controller-01:~/llm-d# kubectl get InferencePool
NAME                      AGE
vllm-llama3-8b-instruct   17m

and 500 response by curl:

root@controller-01:~/llm-d# curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
"model": "food-review",
"prompt": "Write as if you were a critic: San Francisco",
"max_tokens": 100,
"temperature": 0
}'
HTTP/1.1 500 Internal Server Error
date: Thu, 04 Sep 2025 06:29:54 GMT
server: envoy
connection: close
content-length: 0

[nirrozenbaum]

can you add the inf pool cr yaml (running kubectl with -o yaml)

[Frapschen]

@nirrozenbaum

kubectl get InferencePool vllm-llama3-8b-instruct -oyaml

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata:
  annotations:
    meta.helm.sh/release-name: vllm-llama3-8b-instruct
    meta.helm.sh/release-namespace: default
  creationTimestamp: "2025-09-04T06:13:24Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: vllm-llama3-8b-instruct-epp
    app.kubernetes.io/version: v0.3.0
  name: vllm-llama3-8b-instruct
  namespace: default
  resourceVersion: "64214"
  uid: b301a683-f3d2-40b0-a0f8-7e88ee22b2c2
spec:
  extensionRef:
    failureMode: FailClose
    group: ""
    kind: Service
    name: vllm-llama3-8b-instruct-epp
  selector:
    app: vllm-llama3-8b-instruct
  targetPortNumber: 8000
status:
  parent:
  - conditions:
    - lastTransitionTime: "2025-09-04T07:03:06Z"
      message: InferencePool has been accepted by controller kgateway.dev/kgateway
      observedGeneration: 1
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-09-04T07:03:06Z"
      message: All InferencePool references have been resolved
      observedGeneration: 1
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: inference-gateway
      namespace: default

[learner0810]

Maybe you can try adding port to http backendRefs

  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: vllm-llama3-8b-instruct
      port: 8000
      weight: 1

[Frapschen]

@learner0810 applied, and HTTPRoute CR status seems fine.

root@controller-01:~/llm-d# kubectl get httproute llm-route -o yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"gateway.networking.k8s.io/v1","kind":"HTTPRoute","metadata":{"annotations":{},"name":"llm-route","namespace":"default"},"spec":{"parentRefs":[{"group":"gateway.networking.k8s.io","kind":"Gateway","name":"inference-gateway"}],"rules":[{"backendRefs":[{"group":"inference.networking.x-k8s.io","kind":"InferencePool","name":"vllm-llama3-8b-instruct","port":8000}],"matches":[{"path":{"type":"PathPrefix","value":"/"}}],"timeouts":{"request":"300s"}}]}}
  creationTimestamp: "2025-09-04T06:14:44Z"
  generation: 2
  name: llm-route
  namespace: default
  resourceVersion: "75182"
  uid: 6b2bade6-ecf7-4dbf-9f1f-d2ba6c836cf1
spec:
  parentRefs:
  - group: gateway.networking.k8s.io
    kind: Gateway
    name: inference-gateway
  rules:
  - backendRefs:
    - group: inference.networking.x-k8s.io
      kind: InferencePool
      name: vllm-llama3-8b-instruct
      port: 8000
      weight: 1
    matches:
    - path:
        type: PathPrefix
        value: /
    timeouts:
      request: 300s
status:
  parents:
  - conditions:
    - lastTransitionTime: "2025-09-04T06:14:44Z"
      message: ""
      observedGeneration: 2
      reason: Accepted
      status: "True"
      type: Accepted
    - lastTransitionTime: "2025-09-04T07:03:05Z"
      message: ""
      observedGeneration: 2
      reason: ResolvedRefs
      status: "True"
      type: ResolvedRefs
    controllerName: kgateway.dev/kgateway
    parentRef:
      group: gateway.networking.k8s.io
      kind: Gateway
      name: inference-gateway

However, the curl gets 404:

root@controller-01:~/llm-d# curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
"model": "food-review",
"prompt": "Write as if you were a critic: San Francisco",
"max_tokens": 100,
"temperature": 0
}'
HTTP/1.1 404 Not Found
date: Thu, 04 Sep 2025 09:26:38 GMT
server: envoy
content-length: 0

[nirrozenbaum]

sounds like one issue solved.
let me try to help you solve that one as well.
which model server were you using? GPU/CPU/Sim?

[nirrozenbaum]

I'm guessing by analyzing the error that you used either the simulator deployment or the GPU deployment.
can you try to run the same curl but replace the model name to food-review-1?

curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
"model": "food-review-1",
"prompt": "Write as if you were a critic: San Francisco",
"max_tokens": 100,
"temperature": 0
}'

[Frapschen]

sounds like one issue solved. let me try to help you solve that one as well. which model server were you using? GPU/CPU/Sim?

I'm use Sim, I found the epp pod error log:

2025-09-04T09:39:21Z	ERROR	handlers/streamingserver.go:238	Failed to process request	{"request": "request_body:{end_of_stream:true} metadata_context:{}", "error": "inference gateway: BadConfiguration - error finding a model object in InferenceModel for input food-review-1"}
sigs.k8s.io/gateway-api-inference-extension/pkg/epp/handlers.(*StreamingServer).Process
	/src/pkg/epp/handlers/streamingserver.go:238
github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3._ExternalProcessor_Process_Handler
	/go/pkg/mod/github.com/envoyproxy/go-control-plane/envoy@v1.32.4/service/ext_proc/v3/external_processor_grpc.pb.go:106
google.golang.org/grpc.(*Server).processStreamingRPC
	/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1695
google.golang.org/grpc.(*Server).handleStream
	/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1819
google.golang.org/grpc.(*Server).serveStreams.func2.1
	/go/pkg/mod/google.golang.org/grpc@v1.71.0/server.go:1035

I'm trying to add an InferenceModel:

apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferenceModel
metadata:
  name: food-review-1
spec:
  modelName: food-review-1
  criticality: Standard
  poolRef:
    name: vllm-llama3-1b-instruct
  targetModels:
  - name: food-review-1
    weight: 100

but still reponse 404 code

[nirrozenbaum]

just saw there is a bug in the guide that it points to an old version (v0.3.0).
you don't need the InferenceModel CR.

can you please retry to run the same using v0.5.1 in the helm chart?
(and use food-review-1 in the model field)

[Frapschen]

@nirrozenbaum @learner0810 Thank you all, now everything is fine:

root@controller-01:~/llm-d# curl -i ${IP}:${PORT}/v1/completions -H 'Content-Type: application/json' -d '{
"model": "food-review-1",
"prompt": "Write as if you were a critic: San Francisco",
"max_tokens": 100,
"temperature": 0
}'
HTTP/1.1 200 OK
x-envoy-upstream-service-time: 0
x-went-into-resp-headers: true
server: envoy
date: Thu, 04 Sep 2025 09:48:46 GMT
content-type: application/json
transfer-encoding: chunked

{"choices":[{"finish_reason":"stop","index":0,"text":"The rest is silence. "}],"created":1756979326,"do_remote_decode":false,"do_remote_prefill":false,"id":"chatcmpl-e37b8ea0-8c1b-4321-80dd-a7b2deb0d0d2","model":"food-review-1","object":"text_completion","remote_block_ids":null,"remote_engine_id":"","remote_host":"","remote_port":0,"usage":{"completion_tokens":5,"prompt_tokens":9,"total_tokens":14}}

[nirrozenbaum]

perfect!
we have few opened PRs to update the quickstart guide. I hope we get all these issues fixed in the guide asap.