Missing :te header when using envoy proxy with grpc-web filter

[zeionara]

Description
When using envoy proxy to filter grpc requests, the error Missing :te header is thrown by the triton server

Triton Information
What version of Triton are you using? 24.01

Are you using the Triton container or did you build it yourself? I am using pre-built triton container.

To Reproduce
Steps to reproduce the behavior.

1. Configure envoy according to the example;
2. Start triton server and envoy proxy;
3. Send grpc request to the envoy proxy.

Expected behavior

The triton server successfully handles the request redirected by the envoy proxy and returns the result, generated by the model.

Full error text:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/zeio/triton-tutorial/tt/__main__.py", line 167, in <module>
    main()
  File "/home/zeio/.conda/envs/triton-tutorial/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeio/.conda/envs/triton-tutorial/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/home/zeio/.conda/envs/triton-tutorial/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeio/.conda/envs/triton-tutorial/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeio/.conda/envs/triton-tutorial/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/zeio/triton-tutorial/tt/__main__.py", line 43, in embed
    embedding = client.embed(text)
                ^^^^^^^^^^^^^^^^^^
  File "/home/zeio/triton-tutorial/tt/GrpcClient.py", line 40, in embed
    response = self.client.infer(
               ^^^^^^^^^^^^^^^^^^
  File "/home/zeio/.local/lib/python3.11/site-packages/tritonclient/grpc/_client.py", line 1572, in infer
    raise_error_grpc(rpc_error)
  File "/home/zeio/.local/lib/python3.11/site-packages/tritonclient/grpc/_utils.py", line 77, in raise_error_grpc
    raise get_error_grpc(rpc_error) from None
tritonclient.utils.InferenceServerException: [StatusCode.UNKNOWN] Missing :te header

[yinggeh]

@zeionara Hi. Can you reproduce without the envoy proxy?

[zeionara]

@zeionara Hi. Can you reproduce without the envoy proxy?

Not really. Actually I've solved the issue by adding lua script which sets the te header explicitly:

               http_filters:
                 - name: envoy.filters.http.lua
                   typed_config:
                     "@type": type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua
                     inline_code: |
                       function envoy_on_request(request_handle)
                         request_handle:headers():add("te", "trailers")
                       end
                 - name: envoy.filters.http.router
                   typed_config:
                     "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

Does it sound reasonable to make te header optional in the triton server?

[yinggeh]

@jbkyang-nvi Any thoughts on that?

[Pascualex]

We are also experiencing this. I only write (instead of just subscribing) because it seems like a big coincidence that this is from just 2 days ago, but maybe that's just what it is.

[yinggeh]

@zeionara @Pascualex I will bring this up to the team. Thanks!

[fvdnabee]

envoy users might be interested in this change: envoyproxy/envoy#32255

E.g. adding sanitize_te: false to your envoy config should help until envoy releases the fix for 1.29.x:

layered_runtime:
  - name: static_layer_sanitize_te
    static_layer:
      envoy.reloadable_features.sanitize_te: false

[yinggeh]

DLIS-6301

[quantumsheep]

Hey! envoyproxy/envoy#32255 should fix this, we also had the same issue with 1.29.x breaking some of our gRPC services

[ekarmazin]

For my envoy version 1.29.1 this worked:

layered_runtime:
  layers:
    - name: static_layer_sanitize_te
      static_layer:
        envoy.reloadable_features.sanitize_te: false

[zeionara]

envoy users might be interested in this change: envoyproxy/envoy#32255

E.g. adding sanitize_te: false to your envoy config should help until envoy releases the fix for 1.29.x:

layered_runtime:
  - name: static_layer_sanitize_te
    static_layer:
      envoy.reloadable_features.sanitize_te: false

For my envoy version 1.29.1 this worked:

layered_runtime:
  layers:
    - name: static_layer_sanitize_te
      static_layer:
        envoy.reloadable_features.sanitize_te: false

Also worked for me

[oandreeva-nv]

Resolution: envoyproxy/envoy#32255

[nnshah1]

Resolution: envoyproxy/envoy#32255

@whoisj - in case this is relevant for our K8s guides in the future.