HTTPInfo::unmarshal memory leak

[oxwangfeng]

I running trafficserver on ubuntu14.04，I use the new trafficserver，when i use asan Detect memory leaks，I have find many HTTPInfo::unmarshal memory leak；

=================================================================
==32431==ERROR: LeakSanitizer: detected memory leaks

Direct leak of 10752 byte(s) in 21 object(s) allocated from:
#0 0x7f5458d584fa in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x944fa)
#1 0x7f5458a4d49d in ats_malloc /root/ats/trafficserver-master/lib/ts/ink_memory.cc:59
#2 0x8585e7 in HTTPInfo::unmarshal(char*, int, RefCountObj*) /root/ats/trafficserver-master/proxy/hdrs/HTTP.cc:2062
#3 0x90242d in unmarshal_helper /root/ats/trafficserver-master/iocore/cache/Cache.cc:2175
#4 0x9045f2 in CacheVC::handleReadDone(int, Event*) /root/ats/trafficserver-master/iocore/cache/Cache.cc:2383
#5 0x52b242 in Continuation::handleEvent(int, void*) /root/ats/trafficserver-master/iocore/eventsystem/I_Continuation.h:153
#6 0x946b04 in Cache::open_read(Continuation*, ats::CryptoHash const*, HTTPHdr*, OverridableHttpConfigParams*, CacheFragType, char const*, int) /root/ats/trafficserver-master/iocore/cache/CacheRead.cc:162
#7 0x90f63c in CacheProcessor::open_read(Continuation*, HttpCacheKey const*, HTTPHdr*, OverridableHttpConfigParams*, long, CacheFragType) /root/ats/trafficserver-master/iocore/cache/Cache.cc:3281
#8 0x64a2e0 in HttpCacheSM::do_cache_open_read(HttpCacheKey const&) /root/ats/trafficserver-master/proxy/http/HttpCacheSM.cc:253
#9 0x64a708 in HttpCacheSM::open_read(HttpCacheKey const*, URL*, HTTPHdr*, OverridableHttpConfigParams*, long) /root/ats/trafficserver-master/proxy/http/HttpCacheSM.cc:286
#10 0x696496 in HttpSM::do_cache_lookup_and_read() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:4474
#11 0x6b1b07 in HttpSM::set_next_state() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7293
#12 0x6afcd2 in HttpSM::call_transact_and_set_next_state(void ()(HttpTransact::State)) /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7117
#13 0x676196 in HttpSM::handle_api_return() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:1548
#14 0x6c260f in HttpSM::do_api_callout() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:355
#15 0x6afeab in HttpSM::set_next_state() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7150
#16 0x6afcd2 in HttpSM::call_transact_and_set_next_state(void ()(HttpTransact::State)) /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7117
#17 0x6b01bd in HttpSM::set_next_state() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7170
#18 0x6afcd2 in HttpSM::call_transact_and_set_next_state(void ()(HttpTransact::State)) /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7117
#19 0x676196 in HttpSM::handle_api_return() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:1548
#20 0x6c260f in HttpSM::do_api_callout() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:355
#21 0x6afeab in HttpSM::set_next_state() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7150
#22 0x6afcd2 in HttpSM::call_transact_and_set_next_state(void ()(HttpTransact::State)) /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7117
#23 0x676196 in HttpSM::handle_api_return() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:1548
#24 0x6c260f in HttpSM::do_api_callout() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:355
#25 0x6afeab in HttpSM::set_next_state() /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7150
#26 0x6afcd2 in HttpSM::call_transact_and_set_next_state(void ()(HttpTransact::State)) /root/ats/trafficserver-master/proxy/http/HttpSM.cc:7117
#27 0x66f356 in HttpSM::state_read_client_request_header(int, void*) /root/ats/trafficserver-master/proxy/http/HttpSM.cc:733
#28 0x6808c9 in HttpSM::main_handler(int, void*) /root/ats/trafficserver-master/proxy/http/HttpSM.cc:2530
#29 0x52b242 in Continuation::handleEvent(int, void*) /root/ats/trafficserver-master/iocore/eventsystem/I_Continuation.h:153

Direct leak of 3072 byte(s) in 6 object(s) allocated from:
#0 0x7f5458d584fa in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x944fa)
#1 0x7f5458a4d49d in ats_malloc /root/ats/trafficserver-master/lib/ts/ink_memory.cc:59
#2 0x8585e7 in HTTPInfo::unmarshal(char*, int, RefCountObj*) /root/ats/trafficserver-master/proxy/hdrs/HTTP.cc:2062
#3 0x90242d in unmarshal_helper /root/ats/trafficserver-master/iocore/cache/Cache.cc:2175
#4 0x9045f2 in CacheVC::handleReadDone(int, Event*) /root/ats/trafficserver-master/iocore/cache/Cache.cc:2383
#5 0x52b242 in Continuation::handleEvent(int, void*) /root/ats/trafficserver-master/iocore/eventsystem/I_Continuation.h:153
#6 0x910d24 in AIOCallbackInternal::io_complete(int, void*) /root/ats/trafficserver-master/iocore/aio/P_AIO.h:113
#7 0x52b242 in Continuation::handleEvent(int, void*) /root/ats/trafficserver-master/iocore/eventsystem/I_Continuation.h:153
#8 0xa496e3 in EThread::process_event(Event*, int) /root/ats/trafficserver-master/iocore/eventsystem/UnixEThread.cc:122
#9 0xa49f52 in EThread::execute() /root/ats/trafficserver-master/iocore/eventsystem/UnixEThread.cc:188
#10 0xa479c0 in spawn_thread_internal /root/ats/trafficserver-master/iocore/eventsystem/Thread.cc:91
#11 0x7f545781a183 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x8183)

Direct leak of 1792 byte(s) in 4 object(s) allocated from:
#0 0x7f5458d584fa in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x944fa)
#1 0x7f5458a4d49d in ats_malloc /root/ats/trafficserver-master/lib/ts/ink_memory.cc:59
#2 0x4fbd94 in DefaultAlloc::alloc(int) ../../lib/ts/defalloc.h:34
#3 0x9d1999 in Vec<MapElem<char const*, long>, DefaultAlloc, 2>::set_expand() ../../lib/ts/Vec.h:781
#4 0x9d0ebd in HashMap<char const*, StringHashFns, long, DefaultAlloc>::put(char const*, long) ../../lib/ts/Map.h:552
#5 0x9c7266 in SSLInitializeStatistics() /root/ats/trafficserver-master/iocore/net/SSLUtils.cc:1051
#6 0x9ae807 in SSLNetProcessor::start(int, unsigned long) /root/ats/trafficserver-master/iocore/net/SSLNetProcessor.cc:72
#7 0x575d8e in main /root/ats/trafficserver-master/proxy/Main.cc:1833
#8 0x7f5456a3af44 in __libc_start_main (/lib/x86_64-linux-gnu/libc.so.6+0x21f44)

SUMMARY: AddressSanitizer: 15616 byte(s) leaked in 31 allocation(s).

bin/traffic_server -V
traffic_server: using root directory '/opt/fusion/cdn/trafficserver'
Apache Traffic Server - traffic_server - 8.0.0 - (build # 082817 on Aug 28 2017 at 17:30:28)

./configure --prefix=/opt/fusion/cdn/trafficserver --enable-debug CXXFLAGS="-fno-omit-frame-pointer -fsanitize=address -fuse-ld=gold "

Has anyone solved this yet?

[webdriverrr]

Hello @oxwangfeng,

Would you mind to expose a little bit more info about how did you configure trafficserver and how ran it.
I am starting my little investigation regarding this project and would like to stick to stability issues first like yours.

P.S. I have run it in the same manner configured as froward proxy and didn't get such memory leaks, but got another memory leaks in few other places.

[oxwangfeng]

@webdriverrr i running ats on ubuntu 14.04.。。i use the default configure; when i use asan to detect ats 5.3.2,6.0,6.2,7.0,and the newest,and I find the same leak；It's possible that we're leaking fragment tables.
when I turn on proxy.config.dump_mem_info_frequency, iobuffer is rising until memory is Exhausted；

[webdriverrr]

@oxwangfeng so nothing is changed in configs while running TS? Are they default?
BTW do you install TS from sources or deb package?

[oxwangfeng]

@webdriverrr I install ats from source which download from github;

[oxwangfeng]

@webdriverrr Only in large documents will appear.i test ats use .tar.gz file;

[WolfgangBai]

I have the same problem as him.I install ats from source version 6.2.1 which download from github. i found at proxy/hdrs/HTTP.cc func HTTPInfo::unmarshal() allocated memory was not released.

[zwoop]

This is running with freelist disabled I assume?

[bryancall]

If this is still an issue please reopen.

[masaori335]

For the record, below PRs fixed this issue.

• Avoid ats_malloc in unmarshal #4874 fix the memory leak
• Fixed the compatibility with previous cache verison #5075 fix the cache incompatibility made by Avoid ats_malloc in unmarshal #4874
• fix crash in CacheVC::openReadFromWriter #5122 fix crash introduced by Avoid ats_malloc in unmarshal #4874