Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ECDS based Wasm filter configuration causes proxy segmentation fault #35267

Closed
3 of 15 tasks
fatedier opened this issue Sep 18, 2021 · 15 comments
Closed
3 of 15 tasks

ECDS based Wasm filter configuration causes proxy segmentation fault #35267

fatedier opened this issue Sep 18, 2021 · 15 comments
Assignees
@kyessenov
Labels
area/extensions and telemetry lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically.

Comments

@fatedier
Copy link
Contributor

Bug Description

istio-ingressgateway got segmentation fault error when i test wasm extension.

Note that it's occasional and i can't reproduce it with the same steps.

Test steps:

# Delete wasm envoyfilter
kubectl delete -f ./ev.yaml

# Send request to istio-ingressgateway
curl http://x.x.x.x/test

# Reapply wasm envoyfilter
kubectl apply -f ./ev.yaml

# Send request to istio-ingressgateway，connection closed and istio-ingressgateway restart
curl http://x.x.x.x/test

ev.yaml

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: ingressgateway-real-ip
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      app: istio-ingressgateway
  configPatches:
    - applyTo: EXTENSION_CONFIG
      match:
        context: GATEWAY
      patch:
        operation: ADD
        value:
          name: realip
          typed_config:
            '@type': type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
            value:
              config:
                configuration:
                  '@type': type.googleapis.com/google.protobuf.StringValue
                  value: |
                    10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
                vm_config:
                  vm_id: realip
                  code:
                    local:
                      filename: /wasm-filters/realip.wasm
                  runtime: envoy.wasm.runtime.v8
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
        listener:
          filterChain:
            filter:
              name: envoy.http_connection_manager
      patch:
        operation: INSERT_BEFORE
        value:
          name: realip
          config_discovery:
            config_source:
              ads: {}
              initial_fetch_timeout: 0s
            type_urls: [ "type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm"]
Last logs from istio-ingressgateway pod 
2021-09-18T04:05:00.699261Z     critical        envoy backtrace Caught Segmentation fault, suspect faulting address 0x0
2021-09-18T04:05:00.699294Z     critical        envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers):
2021-09-18T04:05:00.699302Z     critical        envoy backtrace Envoy version: 6a2597296110a21043af384282539739d996c0ad/1.18.4-dev/Clean/RELEASE/BoringSSL
2021-09-18T04:05:00.699582Z     critical        envoy backtrace #0: __restore_rt [0x7f9e73626980]
2021-09-18T04:05:00.709743Z     critical        envoy backtrace #1: Envoy::Filter::Http::FilterConfigSubscription::onConfigUpdate() [0x565162db0045]
2021-09-18T04:05:00.718959Z     critical        envoy backtrace #2: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x565162e36579]
2021-09-18T04:05:00.727856Z     critical        envoy backtrace #3: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x565162e3bef4]
2021-09-18T04:05:00.736794Z     critical        envoy backtrace #4: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x565162b126a1]
2021-09-18T04:05:00.747694Z     critical        envoy backtrace #5: Envoy::Grpc::AsyncStreamImpl::onData() [0x565162e506d7]
2021-09-18T04:05:00.757142Z     critical        envoy backtrace #6: Envoy::Http::AsyncStreamImpl::encodeData() [0x565162e55ce2]
2021-09-18T04:05:00.767485Z     critical        envoy backtrace #7: Envoy::Router::UpstreamRequest::decodeData() [0x565162eccef1]
2021-09-18T04:05:00.780015Z     critical        envoy backtrace #8: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x565162c9bf5b]
2021-09-18T04:05:00.791974Z     critical        envoy backtrace #9: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x565162e07bbe]
2021-09-18T04:05:00.803042Z     critical        envoy backtrace #10: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_17::__invoke() [0x565162e0fdc0]
2021-09-18T04:05:00.815230Z     critical        envoy backtrace #11: nghttp2_session_on_data_received [0x5651631472dc]
2021-09-18T04:05:00.826139Z     critical        envoy backtrace #12: nghttp2_session_mem_recv [0x565163149256]
2021-09-18T04:05:00.836016Z     critical        envoy backtrace #13: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x565162e064dd]
2021-09-18T04:05:00.844793Z     critical        envoy backtrace #14: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x565162e071a5]
2021-09-18T04:05:00.854912Z     critical        envoy backtrace #15: Envoy::Http::CodecClient::onData() [0x565162cfc540]
2021-09-18T04:05:00.863853Z     critical        envoy backtrace #16: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x565162cfd6a5]
2021-09-18T04:05:00.872932Z     critical        envoy backtrace #17: Envoy::Network::FilterManagerImpl::onContinueReading() [0x565162f8ac8f]
2021-09-18T04:05:00.881297Z     critical        envoy backtrace #18: Envoy::Network::ConnectionImpl::onReadReady() [0x565162f81539]
2021-09-18T04:05:00.890250Z     critical        envoy backtrace #19: Envoy::Network::ConnectionImpl::onFileEvent() [0x565162f7f0ef]
2021-09-18T04:05:00.898716Z     critical        envoy backtrace #20: std::__1::__function::__func<>::operator()() [0x565162ad3ec1]
2021-09-18T04:05:00.908190Z     critical        envoy backtrace #21: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x565162ad562c]
2021-09-18T04:05:00.918072Z     critical        envoy backtrace #22: event_process_active_single_queue [0x56516315e9d8]
2021-09-18T04:05:00.926834Z     critical        envoy backtrace #23: event_base_loop [0x56516315d3ae]
2021-09-18T04:05:00.935790Z     critical        envoy backtrace #24: Envoy::Server::InstanceImpl::run() [0x565162abaefe]
2021-09-18T04:05:00.944124Z     critical        envoy backtrace #25: Envoy::MainCommonBase::run() [0x565161338644]
2021-09-18T04:05:00.952473Z     critical        envoy backtrace #26: Envoy::MainCommon::main() [0x565161338e36]
2021-09-18T04:05:00.960819Z     critical        envoy backtrace #27: main [0x565161336e7c]
2021-09-18T04:05:00.960988Z     critical        envoy backtrace #28: __libc_start_main [0x7f9e73244bf7]
AsyncClient 0x5651650dbe00, stream_id_: 1449385254900746208
&stream_info_:
 StreamInfoImpl 0x5651650dbfc0, protocol_: 1, response_code_: null, response_code_details_: null, health_check_request_: 0, route_name_:
Http2::ConnectionImpl 0x5651650e6390, max_headers_kb_: 60, max_headers_count_: 100, per_stream_buffer_limit_: 268435456, allow_metadata_: 0, stream_error_on_invalid_http_messaging_: 0, is_outbound_flood_monitored_control_frame_: 0, skip_encoding_empty_trailers_: 1, dispatching_: 1, raised_goaway_: 0, pending_deferred_reset_streams_.size(): 0
&protocol_constraints_:
 ProtocolConstraints 0x5651650e63f0, outbound_frames_: 0, max_outbound_frames_: 10000, outbound_control_frames_: 0, max_outbound_control_frames_: 1000, consecutive_inbound_frames_with_empty_payload_: 0, max_consecutive_inbound_frames_with_empty_payload_: 1, opened_streams_: 1, inbound_priority_frames_: 0, max_inbound_priority_frames_per_stream_: 100, inbound_window_update_frames_: 14, outbound_data_frames_: 14, max_inbound_window_update_frames_per_data_frame_sent_: 10
Number of active streams: 1, current_stream_id_: 1 Dumping current stream:
stream:
 ConnectionImpl::StreamImpl 0x5651650c7000, stream_id_: 1, unconsumed_bytes_: 0, read_disable_count_: 0, local_end_stream_sent_: 0, remote_end_stream_: 0, data_deferred_: 0, received_noninformational_headers_: 1, pending_receive_buffer_high_watermark_called_: 0, pending_send_buffer_high_watermark_called_: 0, reset_due_to_messaging_error_: 0, cookies_:   pending_trailers_to_encode_:   null
 absl::get<ResponseHeaderMapPtr>(headers_or_trailers_):   null
Dumping corresponding downstream request for upstream stream 1:
 UpstreamRequest 0x5651650ec000
2021-09-18T04:05:00.965294Z     error   Epoch 0 exited with error: signal: segmentation fault
2021-09-18T04:05:00.965317Z     info    No more active epochs, terminating

Version

control plane version: 1.10.4
data plane version: 1.10.4

K8s version: 1.16

Additional Information

No response

Affected product area

  • Docs
  • Installation
  • Networking
  • Performance and Scalability
  • Extensions and Telemetry
  • Security
  • Test and Release
  • User Experience
  • Developer Infrastructure
  • Upgrade
  • Multi Cluster
  • Virtual Machine
  • Control Plane Revisions

Is this the right place to submit this?

  • This is not a security vulnerability
  • This is not a question about how to use Istio
@bianpengyuan
Copy link
Contributor

Can you correlate crash with any config update, like EnvoyFilter update? It will be hard to track down without reproduce steps. cc @kyessenov

@bianpengyuan bianpengyuan changed the title enovy of istio-ingressgateway occurs segmentation fault ECDS based Wasm filter configuration causes proxy segmentation fault Sep 20, 2021
@fatedier
Copy link
Contributor Author

I wrote test steps above including the whole EnvoyFilter config.

I can't reproduce it by myself. If someone else meet the same problem, we can look at them together.

@bianpengyuan
Copy link
Contributor

bianpengyuan commented Sep 21, 2021 •
edited
Loading

The backtrace suggests it crashes on filter config update, which should happen if EXTENSION_CONFIG (i.e. ingressgateway-real-ip EnvoyFilter) has any update. So you are saying there is no update to the EnvoyFilter resource but proxy crashes?

@fatedier
Copy link
Contributor Author

No update manually.

I delete enovyfilter and send some test requests. Then reapply the yaml file to do the same tests but failed.

@istio-policy-bot istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Dec 21, 2021
@bianpengyuan
Copy link
Contributor

Not stale. I can see this in my env as well. Need to figure out a way to reproduce.

@istio-policy-bot istio-policy-bot removed the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Dec 21, 2021
@bianpengyuan
Copy link
Contributor

I think this is because of envoyproxy/envoy#14930

@douglas-reid
Copy link
Contributor

With the related fixes in envoy upstream, are we in a better place here now?

@kyessenov
Copy link
Contributor

My local reproduction seems to pass but I'd like @bianpengyuan to confirm since he was able to trigger it in prod.

@rahulanand16nov
Copy link

rahulanand16nov commented Mar 14, 2022 •
edited
Loading

Still present on Envoy 1.21.2-dev (Istio-1.13.2)

@istio-policy-bot istio-policy-bot added the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label May 20, 2022
@istio-policy-bot istio-policy-bot added the lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically. label Jun 4, 2022
@fatedier fatedier reopened this Sep 8, 2022
@istio-policy-bot istio-policy-bot removed the lifecycle/stale Indicates a PR or issue hasn't been manipulated by an Istio team member for a while label Sep 8, 2022
@fatedier
Copy link
Contributor Author

fatedier commented Sep 8, 2022 •
edited
Loading

Still present on istio 1.13.7
Envoy version: e0c6f64173cd0db9370e7ad8b1fbcee370a9f0f3/1.21.5/Clean/RELEASE/BoringSSL

When I update wasm filter configration, istio gateway pods all crashed.

2022-09-08T07:28:01.889994Z     critical        envoy backtrace Caught Segmentation fault, suspect faulting address 0x0
2022-09-08T07:28:01.890037Z     critical        envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers):
2022-09-08T07:28:01.890042Z     critical        envoy backtrace Envoy version: e0c6f64173cd0db9370e7ad8b1fbcee370a9f0f3/1.21.5/Clean/RELEASE/BoringSSL
2022-09-08T07:28:01.890303Z     critical        envoy backtrace #0: __restore_rt [0x7f0fa5ca1420]
2022-09-08T07:28:01.899460Z     critical        envoy backtrace #1: Envoy::Filter::FilterConfigSubscription::onConfigUpdate() [0x55e1d1ce7ca3]
2022-09-08T07:28:01.908382Z     critical        envoy backtrace #2: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x55e1d1d8e129]
2022-09-08T07:28:01.917527Z     critical        envoy backtrace #3: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x55e1d1d93eb7]
2022-09-08T07:28:01.926208Z     critical        envoy backtrace #4: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x55e1d1d95ba6]
2022-09-08T07:28:01.934716Z     critical        envoy backtrace #5: Envoy::Grpc::AsyncStreamImpl::onData() [0x55e1d1dc82c5]
2022-09-08T07:28:01.943020Z     critical        envoy backtrace #6: Envoy::Http::AsyncStreamImpl::encodeData() [0x55e1d1dcd6c2]
2022-09-08T07:28:01.951295Z     critical        envoy backtrace #7: Envoy::Router::UpstreamRequest::decodeData() [0x55e1d1e00bc9]
2022-09-08T07:28:01.959942Z     critical        envoy backtrace #8: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x55e1d1b8350b]
2022-09-08T07:28:01.968511Z     critical        envoy backtrace #9: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x55e1d1d54708]
2022-09-08T07:28:01.977042Z     critical        envoy backtrace #10: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_25::__invoke() [0x55e1d1d5e660]
2022-09-08T07:28:01.985656Z     critical        envoy backtrace #11: nghttp2_session_on_data_received [0x55e1d20e31ec]
2022-09-08T07:28:01.994141Z     critical        envoy backtrace #12: nghttp2_session_mem_recv [0x55e1d20e525f]
2022-09-08T07:28:02.002280Z     critical        envoy backtrace #13: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55e1d1d52e90]
2022-09-08T07:28:02.010711Z     critical        envoy backtrace #14: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55e1d1d53bd5]
2022-09-08T07:28:02.018989Z     critical        envoy backtrace #15: Envoy::Http::CodecClient::onData() [0x55e1d1c20910]
2022-09-08T07:28:02.026995Z     critical        envoy backtrace #16: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x55e1d1c21ab5]
2022-09-08T07:28:02.036027Z     critical        envoy backtrace #17: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55e1d1fe824f]
2022-09-08T07:28:02.046409Z     critical        envoy backtrace #18: Envoy::Network::ConnectionImpl::onReadReady() [0x55e1d1fe09e6]
2022-09-08T07:28:02.055543Z     critical        envoy backtrace #19: Envoy::Network::ConnectionImpl::onFileEvent() [0x55e1d1fde3df]
2022-09-08T07:28:02.064443Z     critical        envoy backtrace #20: std::__1::__function::__func<>::operator()() [0x55e1d1fd3c51]
2022-09-08T07:28:02.074518Z     critical        envoy backtrace #21: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x55e1d1fd515c]
2022-09-08T07:28:02.082977Z     critical        envoy backtrace #22: event_process_active_single_queue [0x55e1d20fa220]
2022-09-08T07:28:02.091519Z     critical        envoy backtrace #23: event_base_loop [0x55e1d20f8c31]
2022-09-08T07:28:02.100503Z     critical        envoy backtrace #24: Envoy::Server::InstanceImpl::run() [0x55e1d194a19b]
2022-09-08T07:28:02.109335Z     critical        envoy backtrace #25: Envoy::MainCommonBase::run() [0x55e1d0565264]
2022-09-08T07:28:02.117919Z     critical        envoy backtrace #26: Envoy::MainCommon::main() [0x55e1d05659c6]
2022-09-08T07:28:02.126530Z     critical        envoy backtrace #27: main [0x55e1d0561c3c]
2022-09-08T07:28:02.126699Z     critical        envoy backtrace #28: __libc_start_main [0x7f0fa5abf083]
AsyncClient 0x55e1d43a6d00, stream_id_: 16569796223946643133
&stream_info_:
  StreamInfoImpl 0x55e1d43a6ed0, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: 1, health_check_request_: 0, route_name_:   upstream_info_:
    UpstreamInfoImpl 0x55e1d4730378, upstream_connection_id_: 18774
Http2::ConnectionImpl 0x55e1d42ed4b0, max_headers_kb_: 60, max_headers_count_: 100, per_stream_buffer_limit_: 268435456, allow_metadata_: 0, stream_error_on_invalid_http_messaging_: 0, is_out
bound_flood_monitored_control_frame_: 0, dispatching_: 1, raised_goaway_: 0, pending_deferred_reset_streams_.size(): 0
&protocol_constraints_:
  ProtocolConstraints 0x55e1d42ed530, outbound_frames_: 0, max_outbound_frames_: 10000, outbound_control_frames_: 0, max_outbound_control_frames_: 1000, consecutive_inbound_frames_with_empty_
payload_: 0, max_consecutive_inbound_frames_with_empty_payload_: 1, opened_streams_: 1, inbound_priority_frames_: 0, max_inbound_priority_frames_per_stream_: 100, inbound_window_update_frames
_: 39, outbound_data_frames_: 57, max_inbound_window_update_frames_per_data_frame_sent_: 10
Number of active streams: 1, current_stream_id_: 1 Dumping current stream:
stream:
  ConnectionImpl::StreamImpl 0x55e1d4178400, stream_id_: 1, unconsumed_bytes_: 0, read_disable_count_: 0, local_end_stream_: 0, local_end_stream_sent_: 0, remote_end_stream_: 0, data_deferred
_: 0, received_noninformational_headers_: 1, pending_receive_buffer_high_watermark_called_: 0, pending_send_buffer_high_watermark_called_: 0, reset_due_to_messaging_error_: 0, cookies_:   pen
ding_trailers_to_encode_:   null
  absl::get<ResponseHeaderMapPtr>(headers_or_trailers_):   null
Dumping corresponding downstream request for upstream stream 1:
  UpstreamRequest 0x55e1d472cc00
2022-09-08T07:28:02.246843Z     info    ads     ADS: "@" istio-ingressgateway-b-6fbbb6bd6c-dbf9g.istio-system-2 terminated rpc error: code = Canceled desc = context canceled
2022-09-08T07:28:02.246851Z     info    ads     ADS: "@" istio-ingressgateway-b-6fbbb6bd6c-dbf9g.istio-system-1 terminated rpc error: code = Canceled desc = context canceled
2022-09-08T07:28:02.402659Z     error   Epoch 0 exited with error: signal: segmentation fault (core dumped)
2022-09-08T07:28:02.402694Z     info    No more active epochs, terminating

@kyessenov
Copy link
Contributor

Does it happen on 1.15? There's a crash fix every release of istio, and this stack trace is not specific enough to tell whether it's new or old.

@fatedier
Copy link
Contributor Author

fatedier commented Sep 8, 2022

Does it happen on 1.15? There's a crash fix every release of istio, and this stack trace is not specific enough to tell whether it's new or old.

It's incidental in my pro istio 1.13 environment. Does 1.13.7 include this fix?

I will test it in istio 1.15 cluster in my test environment. But the trouble is I can't reproduce the problem everytime with same steps.

@kyessenov
Copy link
Contributor

I don't think we backport Wasm fixes. We'll need a way reproduce this regardless on a debug build since the stack trace is not very useful.

@fatedier
Copy link
Contributor Author

@kyessenov I test it in istio 1.14.3 and 1.15.0, it seems no problem.

I close this issue since 1.13 will not be supported soon.

@fatedier fatedier mentioned this issue Sep 19, 2022
Closed
16 tasks
@LugiaChang
Copy link

Still present on istio 1.13.7 Envoy version: e0c6f64173cd0db9370e7ad8b1fbcee370a9f0f3/1.21.5/Clean/RELEASE/BoringSSL

When I update wasm filter configration, istio gateway pods all crashed.

2022-09-08T07:28:01.889994Z     critical        envoy backtrace Caught Segmentation fault, suspect faulting address 0x0
2022-09-08T07:28:01.890037Z     critical        envoy backtrace Backtrace (use tools/stack_decode.py to get line numbers):
2022-09-08T07:28:01.890042Z     critical        envoy backtrace Envoy version: e0c6f64173cd0db9370e7ad8b1fbcee370a9f0f3/1.21.5/Clean/RELEASE/BoringSSL
2022-09-08T07:28:01.890303Z     critical        envoy backtrace #0: __restore_rt [0x7f0fa5ca1420]
2022-09-08T07:28:01.899460Z     critical        envoy backtrace #1: Envoy::Filter::FilterConfigSubscription::onConfigUpdate() [0x55e1d1ce7ca3]
2022-09-08T07:28:01.908382Z     critical        envoy backtrace #2: Envoy::Config::GrpcSubscriptionImpl::onConfigUpdate() [0x55e1d1d8e129]
2022-09-08T07:28:01.917527Z     critical        envoy backtrace #3: Envoy::Config::GrpcMuxImpl::onDiscoveryResponse() [0x55e1d1d93eb7]
2022-09-08T07:28:01.926208Z     critical        envoy backtrace #4: Envoy::Grpc::AsyncStreamCallbacks<>::onReceiveMessageRaw() [0x55e1d1d95ba6]
2022-09-08T07:28:01.934716Z     critical        envoy backtrace #5: Envoy::Grpc::AsyncStreamImpl::onData() [0x55e1d1dc82c5]
2022-09-08T07:28:01.943020Z     critical        envoy backtrace #6: Envoy::Http::AsyncStreamImpl::encodeData() [0x55e1d1dcd6c2]
2022-09-08T07:28:01.951295Z     critical        envoy backtrace #7: Envoy::Router::UpstreamRequest::decodeData() [0x55e1d1e00bc9]
2022-09-08T07:28:01.959942Z     critical        envoy backtrace #8: Envoy::Http::ResponseDecoderWrapper::decodeData() [0x55e1d1b8350b]
2022-09-08T07:28:01.968511Z     critical        envoy backtrace #9: Envoy::Http::Http2::ConnectionImpl::onFrameReceived() [0x55e1d1d54708]
2022-09-08T07:28:01.977042Z     critical        envoy backtrace #10: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_25::__invoke() [0x55e1d1d5e660]
2022-09-08T07:28:01.985656Z     critical        envoy backtrace #11: nghttp2_session_on_data_received [0x55e1d20e31ec]
2022-09-08T07:28:01.994141Z     critical        envoy backtrace #12: nghttp2_session_mem_recv [0x55e1d20e525f]
2022-09-08T07:28:02.002280Z     critical        envoy backtrace #13: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55e1d1d52e90]
2022-09-08T07:28:02.010711Z     critical        envoy backtrace #14: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x55e1d1d53bd5]
2022-09-08T07:28:02.018989Z     critical        envoy backtrace #15: Envoy::Http::CodecClient::onData() [0x55e1d1c20910]
2022-09-08T07:28:02.026995Z     critical        envoy backtrace #16: Envoy::Http::CodecClient::CodecReadFilter::onData() [0x55e1d1c21ab5]
2022-09-08T07:28:02.036027Z     critical        envoy backtrace #17: Envoy::Network::FilterManagerImpl::onContinueReading() [0x55e1d1fe824f]
2022-09-08T07:28:02.046409Z     critical        envoy backtrace #18: Envoy::Network::ConnectionImpl::onReadReady() [0x55e1d1fe09e6]
2022-09-08T07:28:02.055543Z     critical        envoy backtrace #19: Envoy::Network::ConnectionImpl::onFileEvent() [0x55e1d1fde3df]
2022-09-08T07:28:02.064443Z     critical        envoy backtrace #20: std::__1::__function::__func<>::operator()() [0x55e1d1fd3c51]
2022-09-08T07:28:02.074518Z     critical        envoy backtrace #21: Envoy::Event::FileEventImpl::assignEvents()::$_1::__invoke() [0x55e1d1fd515c]
2022-09-08T07:28:02.082977Z     critical        envoy backtrace #22: event_process_active_single_queue [0x55e1d20fa220]
2022-09-08T07:28:02.091519Z     critical        envoy backtrace #23: event_base_loop [0x55e1d20f8c31]
2022-09-08T07:28:02.100503Z     critical        envoy backtrace #24: Envoy::Server::InstanceImpl::run() [0x55e1d194a19b]
2022-09-08T07:28:02.109335Z     critical        envoy backtrace #25: Envoy::MainCommonBase::run() [0x55e1d0565264]
2022-09-08T07:28:02.117919Z     critical        envoy backtrace #26: Envoy::MainCommon::main() [0x55e1d05659c6]
2022-09-08T07:28:02.126530Z     critical        envoy backtrace #27: main [0x55e1d0561c3c]
2022-09-08T07:28:02.126699Z     critical        envoy backtrace #28: __libc_start_main [0x7f0fa5abf083]
AsyncClient 0x55e1d43a6d00, stream_id_: 16569796223946643133
&stream_info_:
  StreamInfoImpl 0x55e1d43a6ed0, protocol_: 1, response_code_: null, response_code_details_: null, attempt_count_: 1, health_check_request_: 0, route_name_:   upstream_info_:
    UpstreamInfoImpl 0x55e1d4730378, upstream_connection_id_: 18774
Http2::ConnectionImpl 0x55e1d42ed4b0, max_headers_kb_: 60, max_headers_count_: 100, per_stream_buffer_limit_: 268435456, allow_metadata_: 0, stream_error_on_invalid_http_messaging_: 0, is_out
bound_flood_monitored_control_frame_: 0, dispatching_: 1, raised_goaway_: 0, pending_deferred_reset_streams_.size(): 0
&protocol_constraints_:
  ProtocolConstraints 0x55e1d42ed530, outbound_frames_: 0, max_outbound_frames_: 10000, outbound_control_frames_: 0, max_outbound_control_frames_: 1000, consecutive_inbound_frames_with_empty_
payload_: 0, max_consecutive_inbound_frames_with_empty_payload_: 1, opened_streams_: 1, inbound_priority_frames_: 0, max_inbound_priority_frames_per_stream_: 100, inbound_window_update_frames
_: 39, outbound_data_frames_: 57, max_inbound_window_update_frames_per_data_frame_sent_: 10
Number of active streams: 1, current_stream_id_: 1 Dumping current stream:
stream:
  ConnectionImpl::StreamImpl 0x55e1d4178400, stream_id_: 1, unconsumed_bytes_: 0, read_disable_count_: 0, local_end_stream_: 0, local_end_stream_sent_: 0, remote_end_stream_: 0, data_deferred
_: 0, received_noninformational_headers_: 1, pending_receive_buffer_high_watermark_called_: 0, pending_send_buffer_high_watermark_called_: 0, reset_due_to_messaging_error_: 0, cookies_:   pen
ding_trailers_to_encode_:   null
  absl::get<ResponseHeaderMapPtr>(headers_or_trailers_):   null
Dumping corresponding downstream request for upstream stream 1:
  UpstreamRequest 0x55e1d472cc00
2022-09-08T07:28:02.246843Z     info    ads     ADS: "@" istio-ingressgateway-b-6fbbb6bd6c-dbf9g.istio-system-2 terminated rpc error: code = Canceled desc = context canceled
2022-09-08T07:28:02.246851Z     info    ads     ADS: "@" istio-ingressgateway-b-6fbbb6bd6c-dbf9g.istio-system-1 terminated rpc error: code = Canceled desc = context canceled
2022-09-08T07:28:02.402659Z     error   Epoch 0 exited with error: signal: segmentation fault (core dumped)
2022-09-08T07:28:02.402694Z     info    No more active epochs, terminating

You mean update the wasm configuration.value in your envoyfilter example?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kyessenov kyessenov

Labels
area/extensions and telemetry lifecycle/automatically-closed Indicates a PR or issue that has been closed automatically.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@fatedier @kyessenov @rahulanand16nov @douglas-reid @LugiaChang @bianpengyuan @istio-policy-bot