Skip to content

Conversation

@scw00
Copy link
Member

@scw00 scw00 commented Aug 5, 2017

This pr tries to fix the #2323.

@scw00 scw00 added the HTTP/2 label Aug 5, 2017
@scw00 scw00 added this to the 8.0.0 milestone Aug 5, 2017
@scw00 scw00 requested review from masaori335 and zwoop August 5, 2017 03:19
@scw00 scw00 self-assigned this Aug 5, 2017
@zwoop
Copy link
Contributor

zwoop commented Aug 6, 2017

Testing on Docs now.

{
node->parent->children.remove(node);
if (node->queued) {
node->parent->queue->erase(node->entry);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the queue of parent become empty by this operation, the queued flag of parent should also be false.
We should do same operations to parent of parent recursively.
So I think we should walk up the tree to correct queue and queued flag of parents here. WDYT?

tree->reprioritize(1, 7, true);

box.check(node_a->queue->in(node_f->entry), "F should be in A's queue");
// do we need to reset C's queued flag since C's queue is empty ??
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think so. In case for C is reprioritized. IMO, walking up the tree in _change_parent will fix this.

@scw00
Copy link
Member Author

scw00 commented Aug 7, 2017

I'll updated after Docs Test.
If it doesn't work, I think logging the http2_con is a better choice for us to address this case. Since I can't reproduce on my local machine .

@zwoop
Copy link
Contributor

zwoop commented Aug 7, 2017

Still tripping up on ASAN. :-/

==32343==ERROR: AddressSanitizer: heap-use-after-free on address 0x602000033258 at pc 0x0000008e4583 bp 0x2aaaaadfe850 sp 0x2aaaaadfe840
READ of size 8 at 0x602000033258 thread T2 ([ET_NET 7])
    #0 0x8e4582 in Http2DependencyTree<Http2Stream*>::_top(Http2DependencyTree<Http2Stream*>::Node*) /usr/local/src/trafficserver/proxy/http2/Http2DependencyTree.h:306
    #1 0x8e2e64 in Http2DependencyTree<Http2Stream*>::top() /usr/local/src/trafficserver/proxy/http2/Http2DependencyTree.h:319
    #2 0x8da955 in Http2ConnectionState::send_data_frames_depends_on_priority() /usr/local/src/trafficserver/proxy/http2/Http2ConnectionState.cc:1207
    #3 0x8d73bf in Http2ConnectionState::main_event_handler(int, void*) /usr/local/src/trafficserver/proxy/http2/Http2ConnectionState.cc:880
    #4 0x67923b in Continuation::handleEvent(int, void*) /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:153
    #5 0xb81916 in EThread::process_event(Event*, int) /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:122
    #6 0xb82166 in EThread::execute() /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:188
    #7 0xb7fcaf in spawn_thread_internal /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:91
    #8 0x2b9b57c5cdc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
    #9 0x2b9b5867676c in clone (/lib64/libc.so.6+0xf776c)

0x602000033258 is located 8 bytes inside of 16-byte region [0x602000033250,0x602000033260)
freed by thread T2 ([ET_NET 7]) here:
    #0 0x609930 in operator delete(void*) (/opt/ats/bin/traffic_server+0x609930)
    #1 0x8cf8d9 in Http2DependencyTree<Http2Stream*>::Node::~Node() /usr/local/src/trafficserver/proxy/http2/Http2DependencyTree.h:67
    #2 0x8e2bb0 in Http2DependencyTree<Http2Stream*>::remove(Http2DependencyTree<Http2Stream*>::Node*) /usr/local/src/trafficserver/proxy/http2/Http2DependencyTree.h:217
    #3 0x8d993f in Http2ConnectionState::delete_stream(Http2Stream*) /usr/local/src/trafficserver/proxy/http2/Http2ConnectionState.cc:1122
    #4 0x8dae22 in Http2ConnectionState::send_data_frames_depends_on_priority() /usr/local/src/trafficserver/proxy/http2/Http2ConnectionState.cc:1233
    #5 0x8d73bf in Http2ConnectionState::main_event_handler(int, void*) /usr/local/src/trafficserver/proxy/http2/Http2ConnectionState.cc:880
    #6 0x67923b in Continuation::handleEvent(int, void*) /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:153
    #7 0xb81916 in EThread::process_event(Event*, int) /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:122
    #8 0xb82166 in EThread::execute() /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:188
    #9 0xb7fcaf in spawn_thread_internal /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:91
    #10 0x2b9b57c5cdc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)

previously allocated by thread T2 ([ET_NET 7]) here:
    #0 0x6092b0 in operator new(unsigned long) (/opt/ats/bin/traffic_server+0x6092b0)
    #1 0x8e347d in Http2DependencyTree<Http2Stream*>::Node::Node(unsigned int, unsigned int, unsigned int, Http2DependencyTree<Http2Stream*>::Node*, Http2Stream*) /usr/local/src/trafficserver/proxy/http2/Http2DependencyTree.h:61
    #2 0x8e2037 in Http2DependencyTree<Http2Stream*>::add(unsigned int, unsigned int, unsigned int, bool, Http2Stream*) /usr/local/src/trafficserver/proxy/http2/Http2DependencyTree.h:174
    #3 0x8d1f3d in rcv_headers_frame /usr/local/src/trafficserver/proxy/http2/Http2ConnectionState.cc:291
    #4 0x8d78a5 in Http2ConnectionState::main_event_handler(int, void*) /usr/local/src/trafficserver/proxy/http2/Http2ConnectionState.cc:898
    #5 0x67923b in Continuation::handleEvent(int, void*) /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:153
    #6 0x8c7555 in send_connection_event /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:58
    #7 0x8ccf35 in Http2ClientSession::do_complete_frame_read() /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:472
    #8 0x8cd45a in Http2ClientSession::state_process_frame_read(int, VIO*, bool) /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:509
    #9 0x8cbc2a in Http2ClientSession::state_start_frame_read(int, void*) /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:402
    #10 0x8caac2 in Http2ClientSession::main_event_handler(int, void*) /usr/local/src/trafficserver/proxy/http2/Http2ClientSession.cc:307
    #11 0x67923b in Continuation::handleEvent(int, void*) /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:153
    #12 0xb2a0b7 in read_signal_and_update /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:124
    #13 0xb312bd in UnixNetVConnection::readSignalAndUpdate(int) /usr/local/src/trafficserver/iocore/net/UnixNetVConnection.cc:1079
    #14 0xaf2d32 in SSLNetVConnection::net_read_io(NetHandler*, EThread*) /usr/local/src/trafficserver/iocore/net/SSLNetVConnection.cc:597
    #15 0xb161a0 in NetHandler::mainNetEvent(int, Event*) /usr/local/src/trafficserver/iocore/net/UnixNet.cc:497
    #16 0x67923b in Continuation::handleEvent(int, void*) /usr/local/src/trafficserver/iocore/eventsystem/I_Continuation.h:153
    #17 0xb81916 in EThread::process_event(Event*, int) /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:122
    #18 0xb8293c in EThread::execute() /usr/local/src/trafficserver/iocore/eventsystem/UnixEThread.cc:266
    #19 0xb7fcaf in spawn_thread_internal /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:91
    #20 0x2b9b57c5cdc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)

Thread T2 ([ET_NET 7]) created by T0 ([TS_MAIN]) here:
    #0 0x5728c8 in __interceptor_pthread_create (/opt/ats/bin/traffic_server+0x5728c8)
    #1 0xb7f626 in ink_thread_create ../../lib/ts/ink_thread.h:152
    #2 0xb7ff38 in Thread::start(char const*, void*, unsigned long, std::function<void ()> const&) /usr/local/src/trafficserver/iocore/eventsystem/Thread.cc:111
    #3 0xb86ea3 in EventProcessor::spawn_event_threads(int, int, unsigned long) /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:336
    #4 0xb874d3 in EventProcessor::start(int, unsigned long) /usr/local/src/trafficserver/iocore/eventsystem/UnixEventProcessor.cc:385
    #5 0x6c0cda in main /usr/local/src/trafficserver/proxy/Main.cc:1768
    #6 0x2b9b585a0b34 in __libc_start_main (/lib64/libc.so.6+0x21b34)

SUMMARY: AddressSanitizer: heap-use-after-free /usr/local/src/trafficserver/proxy/http2/Http2DependencyTree.h:306 in Http2DependencyTree<Http2Stream*>::_top(Http2DependencyTree<Http2Stream*>::Node*)
Shadow bytes around the buggy address:
  0x0c047fffe5f0: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fa fa
  0x0c047fffe600: fa fa 00 00 fa fa fd fd fa fa fd fd fa fa 00 00
  0x0c047fffe610: fa fa fd fa fa fa fd fd fa fa fd fd fa fa 00 00
  0x0c047fffe620: fa fa fd fa fa fa fd fa fa fa fd fd fa fa fa fa
  0x0c047fffe630: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
=>0x0c047fffe640: fa fa fa fa fa fa fd fd fa fa fd[fd]fa fa fd fa
  0x0c047fffe650: fa fa fd fa fa fa fd fd fa fa fd fd fa fa fd fa
  0x0c047fffe660: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fd fd
  0x0c047fffe670: fa fa fd fa fa fa fd fa fa fa fd fa fa fa fd fd
  0x0c047fffe680: fa fa fd fd fa fa fd fa fa fa fd fd fa fa fd fa
  0x0c047fffe690: fa fa fd fd fa fa fd fa fa fa fd fa fa fa fd fa
Shadow byte legend (one shadow byte represents 8 application bytes):
  Addressable:           00
  Partially addressable: 01 02 03 04 05 06 07
  Heap left redzone:       fa
  Heap right redzone:      fb
  Freed heap region:       fd
  Stack left redzone:      f1
  Stack mid redzone:       f2
  Stack right redzone:     f3
  Stack partial redzone:   f4
  Stack after return:      f5
  Stack use after scope:   f8
  Global redzone:          f9
  Global init order:       f6
  Poisoned by user:        f7
  Container overflow:      fc
  Array cookie:            ac
  Intra object redzone:    bb
  ASan internal:           fe
  Left alloca redzone:     ca
  Right alloca redzone:    cb
==32343==ABORTING

@scw00
Copy link
Member Author

scw00 commented Aug 8, 2017

Hi @masaori335 Can you please review again !

masaori335
masaori335 previously approved these changes Aug 8, 2017
Copy link
Contributor

@masaori335 masaori335 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me 👍
But not sure this will fix the ASan issue:p

@zwoop
Copy link
Contributor

zwoop commented Aug 8, 2017

If this still doesn't fix the ASAN issue, do we still want this for 7.1.1 ?

@zwoop
Copy link
Contributor

zwoop commented Aug 8, 2017

Fwiw, it still triggers on Docs with this :-).

@scw00
Copy link
Member Author

scw00 commented Aug 9, 2017

May be we need to block this pr until I address or reproduce this case.

@zwoop
Copy link
Contributor

zwoop commented Aug 9, 2017

I don't know how to reproduce it, but I was browsing docs.trafficserver from Safari and Chrome at the same time, and it crashed. Not sure if it was me, or someone else though :-).

@zwoop
Copy link
Contributor

zwoop commented Aug 9, 2017

I marked this as WIP for now, so we don't accidentally merge it.

@zwoop zwoop added the WIP label Aug 9, 2017
@scw00
Copy link
Member Author

scw00 commented Aug 9, 2017

👍

@scw00
Copy link
Member Author

scw00 commented Aug 13, 2017

Hi @zwoop @masaori335 Updated ! Can you please review again !!

@scw00
Copy link
Member Author

scw00 commented Aug 13, 2017

Only fixed the conflicts !

@scw00
Copy link
Member Author

scw00 commented Aug 15, 2017

@zwoop I think this pr is ready to launch ! Can you please backport to 7.1.x

@zwoop zwoop removed the WIP label Aug 16, 2017
@zwoop zwoop merged commit 7f3df8c into apache:master Aug 16, 2017
@zwoop
Copy link
Contributor

zwoop commented Aug 16, 2017

Can we hold off on this one to 7.1.2 ? Or do we need it for 7.1.1 ? I'm just about ready to make the 7.1.1 RC, and this has not been tested particularly well.

@zwoop zwoop modified the milestones: 8.0.0, 7.1.2 Sep 29, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants