503 error with websocket

[khaitranhq]

Description

• Setup envoy with websocket has 503 error. I read an issue about this and get a suggestion that's turning on keepalive. However, I can't find this option in the docs

• I tested without envoy (using direct `http://localhost:3001) and it worked.

Update Application return 503

Code

admin:
  access_log_path: /tmp/admin_access.log
  address:
    socket_address: { address: 0.0.0.0, port_value: 8081 }

static_resources:
  listeners:
    - address:
        socket_address:
          address: 0.0.0.0
          port_value: 80
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                stat_prefix: ingress_http
                route_config:
                  virtual_hosts:
                    - name: backend
                      domains: 
                        - "localhost"
                      routes:
                        - match:
                            prefix: "/"
                          redirect:
                            path_redirect: "/"
                            https_redirect: true
                http_filters:
                  - name: envoy.router
                    config: {}
    - name: listener_0
      address:
        socket_address: { address: 0.0.0.0, port_value: 443 }
      filter_chains:
        - filters:
            - name: envoy.http_connection_manager
              config:
                codec_type: auto
                stat_prefix: ingress_ws_to_ws
                upgrade_configs:
                  - upgrade_type: websocket
                route_config:
                  name: local_route
                  virtual_hosts:
                    - name: local_service
                      domains: ["*", "127.0.0.1", "localhost"]
                      routes:
                        - match:
                            prefix: "/"
                          route:
                            upgrade_configs:
                              upgrade_type: websocket
                            cors:
                              allow_credentials: true
                            cluster: ws
                      cors:
                        allow_origin_string_match:
                          - prefix: "*"
                        allow_methods: GET, PUT, DELETE, POST, OPTIONS
                        allow_headers: keep-alive,user-agent,cache-control,content-type,content-transfer-encoding,custom-header-1,x-accept-content-transfer-encoding,x-accept-response-streaming,x-user-agent,x-grpc-web,grpc-timeout
                        max_age: "1728000"
                        expose_headers: custom-header-1,grpc-status,grpc-message
                http_filters:
                  - name: envoy.grpc_web
                  - name: envoy.cors
                  - name: envoy.router
                    config: {}
          transport_socket:
            name: envoy.transport_sockets.tls
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext
              common_tls_context:
                tls_certificates:
                  - certificate_chain:
                      filename: certs/localhost.crt
                    private_key:
                      filename: certs/localhost.key
  clusters:
    - name: ws
      connect_timeout: 0.25s
      type: logical_dns
      http2_protocol_options: {}
      lb_policy: round_robin
      hosts: 
        - socket_address: 
            address: backend
            port_value: 3001

Thanks for any help!

[zuercher]

If you enable access logging and log the %RESPONSE_CODE_DETAILS% field, it may shed light on the problem. e.g., perhaps Envoy cannot connect to to backend:3001 or the error is coming up from the upstream.

Alternatively, turning on debug logging which should log attempts to connect to the upstream, the forwarded request and the upstream's response headers.

[khaitranhq]

@zuercher can you give me an example for getting access logging, I tried to follow docs to configure envoy.yaml but it makes so many errors.

[zuercher]

This configures logging:

envoy/test/config/integration/server.yaml

Line 53 in a12869f

- name: accesslog

It uses templates and is for testing log filter, so change path to an actual path on your system and remove the filters: section.

[khaitranhq]

I received this message

 {"message":"upstream_reset_before_response_started{connection_failure}","status":503}

Do you have any idea? Following #14981, the upstream's connection might be closed. However, I don't think websocket will close the connection at any time.

[khaitranhq]

Thanks for your support. It worked after using upstream_connection_options with

tcp_keepalive: 
  keepalive_time: 1
  keepalive_interval: 10

Following #8639