Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Router watermark crash #1443

Closed
mattklein123 opened this issue Aug 10, 2017 · 1 comment · Fixed by #1439
Closed

Router watermark crash #1443

mattklein123 opened this issue Aug 10, 2017 · 1 comment · Fixed by #1439
Assignees
@alyssawilk
Labels
bug
Milestone
1.4.0

Comments

@mattklein123
Copy link
Member 

Backtrace:
#0  Envoy::Router::Filter::onBelowWriteBufferLowWatermark (this=0x2218b190) at external/envoy/source/common/router/router.cc:308
#0  Envoy::Router::Filter::onBelowWriteBufferLowWatermark (this=0x2218b190) at external/envoy/source/common/router/router.cc:308
#1  0x00000000005167bd in callLowWatermarkCallbacks (this=<optimized out>) at external/envoy/source/common/http/conn_manager_impl.cc:912
#2  Envoy::Http::ConnectionManagerImpl::onBelowWriteBufferLowWatermark (this=0x842f800) at bazel-out/local-opt/bin/external/envoy/source/common/http/_virtual_includes/conn_manager_lib/common/http/conn_manager_impl.h:299
#3  0x00000000005011aa in onLowWatermark (this=0x123d2880) at external/envoy/source/common/network/connection_impl.cc:311
#4  operator() (__closure=<optimized out>) at external/envoy/source/common/network/connection_impl.cc:64
#5  std::_Function_handler<void(), Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::DispatcherImpl&, int, Envoy::Network::Address::InstanceConstSharedPtr, Envoy::Network::Address::InstanceConstSharedPtr, bool)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/5/functional:1871
#6  0x0000000000501030 in Envoy::Ssl::ConnectionImpl::doWriteToSocket (this=this@entry=0x123d2880) at external/envoy/source/common/ssl/connection_impl.cc:213
#7  0x00000000005010bf in Envoy::Ssl::ConnectionImpl::doWriteToSocket (this=0x123d2880) at external/envoy/source/common/ssl/connection_impl.cc:162
#8  0x0000000000501cab in Envoy::Network::ConnectionImpl::onWriteReady (this=0x123d2880) at external/envoy/source/common/network/connection_impl.cc:459
#9  0x00000000004944f8 in operator() (__args#0=2, this=<optimized out>) at /usr/include/c++/5/functional:2267
#10 operator() (__closure=0x0, arg=<optimized out>, what=<optimized out>) at external/envoy/source/common/event/file_event_impl.cc:60
#11 Envoy::Event::FileEventImpl::<lambda(int, short int, void*)>::_FUN(int, short, void *) () at external/envoy/source/common/event/file_event_impl.cc:61
#12 0x00000000006116b2 in event_persist_closure (ev=<optimized out>, base=0x1302840) at event.c:1580
#13 event_process_active_single_queue (base=base@entry=0x1302840, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=<optimized out>) at event.c:1639
#14 0x0000000000611e0f in event_process_active (base=0x1302840) at event.c:1738
#15 event_base_loop (base=0x1302840, flags=0) at event.c:1961
#16 0x0000000000491039 in Envoy::Server::WorkerImpl::threadRoutine (this=0x16daae0, guard_dog=...) at external/envoy/source/server/worker_impl.cc:110
#17 0x000000000061b11e in Envoy::Thread::Thread::Thread(std::function<void ()>)::{lambda(void*)#1}::_FUN(void*) () at /usr/include/c++/5/functional:2267
#18 0x00007f393b17d184 in start_thread (arg=0x7f3939b48700) at pthread_create.c:312
#19 0x00007f393aea9ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
@mattklein123 mattklein123 added this to the 1.4.0 milestone Aug 10, 2017
@mattklein123
Copy link
Member Author

Another dump with different stack:

Backtrace:
#0  Envoy::Router::Filter::onAboveWriteBufferHighWatermark (this=0x1364b210) at external/envoy/source/common/router/router.cc:316
#0  Envoy::Router::Filter::onAboveWriteBufferHighWatermark (this=0x1364b210) at external/envoy/source/common/router/router.cc:316
#1  0x000000000051673d in callHighWatermarkCallbacks (this=<optimized out>) at external/envoy/source/common/http/conn_manager_impl.cc:906
#2  Envoy::Http::ConnectionManagerImpl::onAboveWriteBufferHighWatermark (this=0x27f21500) at bazel-out/local-opt/bin/external/envoy/source/common/http/_virtual_includes/conn_manager_lib/common/http/conn_manager_impl.h:294
#3  0x000000000050122a in onHighWatermark (this=0xe363900) at external/envoy/source/common/network/connection_impl.cc:318
#4  operator() (__closure=<optimized out>) at external/envoy/source/common/network/connection_impl.cc:65
#5  std::_Function_handler<void(), Envoy::Network::ConnectionImpl::ConnectionImpl(Envoy::Event::DispatcherImpl&, int, Envoy::Network::Address::InstanceConstSharedPtr, Envoy::Network::Address::InstanceConstSharedPtr, bool)::<lambda()> >::_M_invoke(const std::_Any_data &) (__functor=...) at /usr/include/c++/5/functional:1871
#6  0x0000000000501524 in Envoy::Network::ConnectionImpl::write (this=0xe363900, data=...) at external/envoy/source/common/network/connection_impl.cc:274
#7  0x000000000052f577 in Envoy::Http::Http2::ConnectionImpl::StreamImpl::onDataSourceSend (this=0x1837a000, framehd=<optimized out>, length=4048) at external/envoy/source/common/http/http2/codec_impl.cc:224
#8  0x0000000000536db4 in session_call_send_data (framebufs=0x14c0f550, item=<optimized out>, session=0x14c0f400) at nghttp2_session.c:2839
#9  nghttp2_session_mem_send_internal (session=session@entry=0x14c0f400, data_ptr=data_ptr@entry=0x7f9c052f0538, fast_cb=fast_cb@entry=0) at nghttp2_session.c:3099
#10 0x000000000053798e in nghttp2_session_send (session=0x14c0f400) at nghttp2_session.c:3206
#11 0x0000000000530adf in Envoy::Http::Http2::ConnectionImpl::sendPendingFrames (this=0x120d36c8) at external/envoy/source/common/http/http2/codec_impl.cc:566
#12 0x000000000051b161 in Envoy::Http::ConnectionManagerImpl::ActiveStream::encodeData (this=0x1fdef980, filter=<optimized out>, data=..., end_stream=<optimized out>) at external/envoy/source/common/http/conn_manager_impl.cc:835
#13 0x0000000000531734 in Envoy::Http::Http2::ConnectionImpl::onFrameReceived (this=0x22f1c008, frame=<optimized out>) at external/envoy/source/common/http/http2/codec_impl.cc:445
#14 0x0000000000534627 in session_call_on_frame_received (frame=0x12e45598, session=0x12e45400) at nghttp2_session.c:3262
#15 nghttp2_session_on_data_received (session=session@entry=0x12e45400, frame=frame@entry=0x12e45598) at nghttp2_session.c:4877
#16 0x0000000000538692 in session_process_data_frame (session=0x12e45400) at nghttp2_session.c:4897
#17 nghttp2_session_mem_recv (session=0x12e45400, in=0x22d5d34f "", inlen=4048) at nghttp2_session.c:6438
#18 0x000000000053100f in Envoy::Http::Http2::ConnectionImpl::dispatch (this=0x22f1c008, data=...) at external/envoy/source/common/http/http2/codec_impl.cc:300
#19 0x00000000004f12bf in Envoy::Http::CodecClient::onData (this=0x281de300, data=...) at external/envoy/source/common/http/codec_client.cc:100
#20 0x00000000004f141d in Envoy::Http::CodecClient::CodecReadFilter::onData (this=<optimized out>, data=...) at bazel-out/local-opt/bin/external/envoy/source/common/http/_virtual_includes/codec_client_lib/common/http/codec_client.h:139
#21 0x0000000000503b29 in Envoy::Network::FilterManagerImpl::onContinueReading (this=this@entry=0x10a6e6e8, filter=filter@entry=0x0) at external/envoy/source/common/network/filter_manager_impl.cc:61
#22 0x0000000000503b9c in Envoy::Network::FilterManagerImpl::onRead (this=this@entry=0x10a6e6e8) at external/envoy/source/common/network/filter_manager_impl.cc:71
#23 0x0000000000501c19 in onRead (read_buffer_size=223725, this=0x10a6e6e0) at external/envoy/source/common/network/connection_impl.cc:197
#24 Envoy::Network::ConnectionImpl::onReadReady (this=0x10a6e6e0) at external/envoy/source/common/network/connection_impl.cc:395
#25 0x0000000000501d83 in Envoy::Network::ConnectionImpl::onFileEvent (this=0x10a6e6e0, events=3) at external/envoy/source/common/network/connection_impl.cc:343
#26 0x00000000004944f8 in operator() (__args#0=3, this=<optimized out>) at /usr/include/c++/5/functional:2267
#27 operator() (__closure=0x0, arg=<optimized out>, what=<optimized out>) at external/envoy/source/common/event/file_event_impl.cc:60
#28 Envoy::Event::FileEventImpl::<lambda(int, short int, void*)>::_FUN(int, short, void *) () at external/envoy/source/common/event/file_event_impl.cc:61
#29 0x00000000006116b2 in event_persist_closure (ev=<optimized out>, base=0x1f3c580) at event.c:1580
#30 event_process_active_single_queue (base=base@entry=0x1f3c580, max_to_process=max_to_process@entry=2147483647, endtime=endtime@entry=0x0, activeq=<optimized out>) at event.c:1639
#31 0x0000000000611e0f in event_process_active (base=0x1f3c580) at event.c:1738
#32 event_base_loop (base=0x1f3c580, flags=0) at event.c:1961
#33 0x0000000000491039 in Envoy::Server::WorkerImpl::threadRoutine (this=0x2314ba0, guard_dog=...) at external/envoy/source/server/worker_impl.cc:110
#34 0x000000000061b11e in Envoy::Thread::Thread::Thread(std::function<void ()>)::{lambda(void*)#1}::_FUN(void*) () at /usr/include/c++/5/functional:2267
#35 0x00007f9c06125184 in start_thread (arg=0x7f9c052f1700) at pthread_create.c:312
#36 0x00007f9c05e51ffd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

I think this is basically the same issue. Downstream flow control after upstream stream is gone. I think the fix you are working on now which removes callbacks will fix this also.

@alyssawilk alyssawilk mentioned this issue Aug 14, 2017
Merged
mattklein123 pushed a commit that referenced this issue Aug 16, 2017
@alyssawilk @mattklein123
Fix for upstream_request_ / request_encoder_ lifetime issues (#1439)
0649639 
This should fix the issue where the router gets a watermark event before it has stats
(onAboveWriteBufferHighWatermark before decodeHeaders())
This can happen on muxed H2 where one stream's response causes a high 
watermark event while another stream is parsing large headers.

The fix also addresses passing on high watermark events to the new downstream 
stream in case of retries.

part of #150
Fixes #1447
Fixes #1443
rshriram pushed a commit to rshriram/envoy that referenced this issue Oct 30, 2018
@qiwzhang
Support per-route-config from Envoy v2 RDS (envoyproxy#1443)
bfca799
jpsim pushed a commit that referenced this issue Nov 28, 2022
@goaway @jpsim
config: update keepalive settings (#1443)
9db78d1 
Signed-off-by: Mike Schore <mike.schore@gmail.com>
Signed-off-by: JP Simard <jp@jpsim.com>
jpsim pushed a commit that referenced this issue Nov 29, 2022
@goaway @jpsim
config: update keepalive settings (#1443)
14b8439 
Signed-off-by: Mike Schore <mike.schore@gmail.com>
Signed-off-by: JP Simard <jp@jpsim.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alyssawilk alyssawilk

Labels
bug
Projects
None yet
Milestone
1.4.0
Development

Successfully merging a pull request may close this issue.

Fix for upstream_request_ / request_encoder_ lifetime issues alyssawilk/envoy
2 participants
@mattklein123 @alyssawilk