Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Envoy 1.17.1 crashes when a ratelimit cluster is configured, but is unavailable #15747

Closed
petedmarsh opened this issue Mar 29, 2021 · 6 comments
Closed

Envoy 1.17.1 crashes when a ratelimit cluster is configured, but is unavailable #15747

petedmarsh opened this issue Mar 29, 2021 · 6 comments
Assignees
@esmet
Labels
bug triage Issue requires triage

Comments

@petedmarsh
Copy link
Contributor

If you are reporting any crash or any potential security issue, do not
open an issue in this repo. Please report the issue via emailing
envoy-security@googlegroups.com where the issue will be triaged appropriately.

Title: Envoy 1.17.1 crashes when a ratelimit cluster is configured, but is unavailable

Description:

I believe there is a regression in Envoy 1.17.x where if ratelimiting is configured for a cluster but the rate limit service is unavailable then a request to that cluster will cause Envoy to crash. This does not happen in 1.16.0.

This was reported to envoy-security@googlegroups.com and triaged and it was determined that an issue could be opened.

Repro steps:

You can find a reproducible demo here: https://github.com/petedmarsh/envoycrashdemo.

Admin and Stats Output:

Include the admin output for the following endpoints: /stats,
/clusters, /routes, /server_info. For more information, refer to the
admin endpoint documentation.

Note: If there are privacy concerns, sanitize the data prior to
sharing.

Config:

admin:
  access_log_path: /dev/null
  address:
    socket_address:
      address: 0.0.0.0
      port_value: 9901

static_resources:
  clusters:
    - name: echo-cluster
      type: STRICT_DNS
      connect_timeout: 1s
      lb_policy: ROUND_ROBIN
      protocol_selection: USE_DOWNSTREAM_PROTOCOL
      http2_protocol_options: {}
      load_assignment:
        cluster_name: echo
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: echo
                      port_value: 80
    #NOTE! envoy-ratelimit isn't running and isn't supposed to be for the demo!
    - name: ratelimit
      connect_timeout: 4s
      type: STRICT_DNS
      lb_policy: ROUND_ROBIN
      protocol_selection: USE_CONFIGURED_PROTOCOL
      http2_protocol_options: {}
      load_assignment:
        cluster_name: "ratelimit"
        endpoints:
          - lb_endpoints:
              - endpoint:
                  address:
                    socket_address:
                      address: 127.0.0.1
                      port_value: 8081
  listeners:
  - name: echo-listener
    address:
      socket_address:
        address: 0.0.0.0
        port_value: 8888
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          access_log:
          - name: envoy.access_loggers.file
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.access_loggers.file.v3.FileAccessLog
              path: /dev/stdout
          http_filters:
            - name: envoy.filters.http.ratelimit
              typed_config:
                "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit"
                domain: "noop_test"
                failure_mode_deny: false
                timeout: 0.1s
                stage: 0
                rate_limit_service:
                  grpc_service:
                    envoy_grpc:
                      cluster_name: ratelimit      
            - name: envoy.filters.http.router
          route_config:
            name: local_route
            virtual_hosts:
            - name: local_service
              domains: ["*"]
              routes:
              - match:
                  prefix: "/"
                route:
                  cluster: echo-cluster

Logs:

envoy_1  | [2021-03-29 16:26:01.607][14][debug][http] [source/common/http/conn_manager_impl.cc:886] [C0][S17663759149523240964] request headers complete (end_stream=true):
envoy_1  | ':authority', 'localhost:8888'
envoy_1  | ':path', '/status/200'
envoy_1  | ':method', 'GET'
envoy_1  | 'user-agent', 'curl/7.54.0'
envoy_1  | 'accept', '*/*'
envoy_1  |
envoy_1  | [2021-03-29 16:26:01.607][14][debug][http] [source/common/http/filter_manager.cc:755] [C0][S17663759149523240964] request end stream
envoy_1  | [2021-03-29 16:26:01.607][14][warning][misc] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/utility_lib/common/config/utility.h:206] V2 (and AUTO) xDS transport protocol versions are deprecated in grpc_service {
envoy_1  |   envoy_grpc {
envoy_1  |     cluster_name: "ratelimit"
envoy_1  |   }
envoy_1  | }
envoy_1  | . The v2 xDS major version is deprecated and disabled by default. Support for v2 will be removed from Envoy at the start of Q1 2021. You may make use of v2 in Q4 2020 by following the advice in https://www.envoyproxy.io/docs/envoy/latest/faq/api/transition.
envoy_1  | [2021-03-29 16:26:01.707][14][critical][main] [source/exe/terminate_handler.cc:13] std::terminate called! (possible uncaught exception, see trace)
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: d6a4496e712d7a2335b26e2f76210d5904002c26/1.17.1/Clean/RELEASE/BoringSSL
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #0: [0x55b61d056dfb]
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #1: [0x55b61d056c69]
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #2: [0x55b61d6e0943]
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #3: [0x55b61b5af206]
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #4: [0x55b61cd38f4d]
envoy_1  | [2021-03-29 16:26:01.707][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #5: [0x55b61cd6266e]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #6: [0x55b61cd4fa34]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #7: [0x55b61cd715a1]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #8: [0x55b61cd6f8d7]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #9: [0x55b61cd6c226]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #10: [0x55b61ced181b]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #11: [0x55b61cd6dfc1]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #12: [0x55b61cd6d88e]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #13: [0x55b61cd75b82]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #14: [0x55b61cea3847]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #15: [0x55b61cd6d3c1]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #16: [0x55b61cd4bd1c]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #17: [0x55b61ca6ebcf]
envoy_1  | [2021-03-29 16:26:01.708][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #18: [0x55b61ca6934d]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #19: [0x55b61ca66ff9]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #20: [0x55b61ca5c311]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #21: [0x55b61ca5d0fc]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #22: [0x55b61cecb4c8]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #23: [0x55b61cec9e9e]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #24: [0x55b61ca4d1f8]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #25: [0x55b61d0978f3]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: start_thread [0x7fa5557b66db]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Aborted, suspect faulting address 0x6500000001
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: d6a4496e712d7a2335b26e2f76210d5904002c26/1.17.1/Clean/RELEASE/BoringSSL
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __restore_rt [0x7fa5557c1980]
envoy_1  | [2021-03-29 16:26:01.709][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #1: [0x55b61d056c69]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #2: [0x55b61d6e0943]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #3: [0x55b61b5af206]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #4: [0x55b61cd38f4d]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #5: [0x55b61cd6266e]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #6: [0x55b61cd4fa34]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #7: [0x55b61cd715a1]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #8: [0x55b61cd6f8d7]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #9: [0x55b61cd6c226]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #10: [0x55b61ced181b]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #11: [0x55b61cd6dfc1]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #12: [0x55b61cd6d88e]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #13: [0x55b61cd75b82]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #14: [0x55b61cea3847]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #15: [0x55b61cd6d3c1]
envoy_1  | [2021-03-29 16:26:01.710][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #16: [0x55b61cd4bd1c]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #17: [0x55b61ca6ebcf]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #18: [0x55b61ca6934d]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #19: [0x55b61ca66ff9]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #20: [0x55b61ca5c311]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #21: [0x55b61ca5d0fc]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #22: [0x55b61cecb4c8]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #23: [0x55b61cec9e9e]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #24: [0x55b61ca4d1f8]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #25: [0x55b61d0978f3]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: start_thread [0x7fa5557b66db]
envoy_1  | ActiveStream 0x54673f7f0300, stream_id_: 17663759149523240964&filter_manager_:
envoy_1  |   FilterManager 0x54673f7f0378, state_.has_continue_headers_: 0
envoy_1  |   filter_manager_callbacks_.requestHeaders():
envoy_1  |     ':authority', 'localhost:8888'
envoy_1  |     ':path', '/status/200'
envoy_1  |     ':method', 'GET'
envoy_1  |     'user-agent', 'curl/7.54.0'
envoy_1  |     'accept', '*/*'
envoy_1  |     'x-forwarded-proto', 'http'
envoy_1  |     'x-request-id', 'fcc88abc-9a5c-4966-9844-8aa60f1b1926'
envoy_1  |   filter_manager_callbacks_.requestTrailers():   null
envoy_1  |   filter_manager_callbacks_.responseHeaders():   null
envoy_1  |   filter_manager_callbacks_.responseTrailers():   null
envoy_1  |   &stream_info_:
envoy_1  |     StreamInfoImpl 0x54673f7f0450, protocol_: 1, response_code_: null, response_code_details_: null, health_check_request_: 0, route_name_:
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:104] Caught Segmentation fault, suspect faulting address 0x0
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:91] Backtrace (use tools/stack_decode.py to get line numbers):
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:92] Envoy version: d6a4496e712d7a2335b26e2f76210d5904002c26/1.17.1/Clean/RELEASE/BoringSSL
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #0: __restore_rt [0x7fa5557c1980]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #1: [0x55b61d056c69]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #2: [0x55b61d6e0943]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #3: [0x55b61b5af206]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #4: [0x55b61cd38f4d]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #5: [0x55b61cd6266e]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #6: [0x55b61cd4fa34]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #7: [0x55b61cd715a1]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #8: [0x55b61cd6f8d7]
envoy_1  | [2021-03-29 16:26:01.711][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #9: [0x55b61cd6c226]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #10: [0x55b61ced181b]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #11: [0x55b61cd6dfc1]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #12: [0x55b61cd6d88e]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #13: [0x55b61cd75b82]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #14: [0x55b61cea3847]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #15: [0x55b61cd6d3c1]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #16: [0x55b61cd4bd1c]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #17: [0x55b61ca6ebcf]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #18: [0x55b61ca6934d]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #19: [0x55b61ca66ff9]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #20: [0x55b61ca5c311]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #21: [0x55b61ca5d0fc]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #22: [0x55b61cecb4c8]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #23: [0x55b61cec9e9e]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #24: [0x55b61ca4d1f8]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:98] #25: [0x55b61d0978f3]
envoy_1  | [2021-03-29 16:26:01.712][14][critical][backtrace] [bazel-out/k8-opt/bin/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:96] #26: start_thread [0x7fa5557b66db]
envoy_1  | Our FatalActions triggered a fatal signal.
envoycrashdemo_envoy_1 exited with code 139

Call Stack:

(I will attempt to get this for you)
@petedmarsh petedmarsh added bug triage Issue requires triage labels Mar 29, 2021
@esmet
Copy link
Contributor

esmet commented Mar 29, 2021

I'll take a look.

I was able to reproduce this locally with Envoy v1.17.0, but not with a build of Envoy based on v1.15.1. Either way, I'm waiting for a recent build from main and then I'll figure out why this is crashing.

@esmet
Copy link
Contributor

esmet commented Mar 29, 2021 •
edited
Loading

The real error is this exception:

envoy_1  | [2021-03-29 16:26:01.607][14][warning][misc] [bazel-out/k8-opt/bin/source/common/config/_virtual_includes/utility_lib/common/config/utility.h:206] V2 (and AUTO) xDS transport protocol versions are deprecated in grpc_service {
envoy_1  |   envoy_grpc {
envoy_1  |     cluster_name: "ratelimit"
envoy_1  |   }
envoy_1  | }

This works for me once I set transport_api_version to "V3":

...
 59             - name: envoy.filters.http.ratelimit
 60               typed_config:
 61                 "@type": "type.googleapis.com/envoy.extensions.filters.http.ratelimit.v3.RateLimit"
 62                 domain: "noop_test"
 63                 failure_mode_deny: false
 64                 timeout: 0.1s
 65                 stage: 0
 66                 rate_limit_service:
 67                   transport_api_version: V3
 68                   grpc_service:
 69                     envoy_grpc:
 70                       cluster_name: ratelimit
 ...

Starting with v1.17, transport_api_version is now required to force transitioning off of V2 APIs.

Let me know if setting this field works for you, too.

@esmet
Copy link
Contributor

esmet commented Mar 29, 2021

/assign

@mattklein123
Copy link
Member

This is a dup of #15083 and fixed by #15548. @htuch it would be really great to land that other PR.

@htuch
Copy link
Member

htuch commented Mar 30, 2021

@mattklein123 I've updated #15548, fixed the fuzz issue.

@petedmarsh
Copy link
Contributor Author

I can confirm that this fixes it for me too, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@esmet esmet

Labels
bug triage Issue requires triage
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@petedmarsh @esmet @mattklein123 @htuch