Skip to content

ATS 9.1.4 memory leak and crashed with signal 6 #9969

@xq2248

Description

@xq2248

ATS build options:
/configure --prefix=/home/zxats --with-user=zxats --with-group=zxats --enable-hwloc --enable-example-plugins --enable-experimental-plugins --with-jemalloc=./jemalloc-5.2.1

Test Case:
The sizes of all the cache objects in orgin server are the same, a lot of 28KB pictures.
The ram_cache_cutoff is set to 1KB, so the cached pictures can be cache in RAM.
With pressure, ATS crashed with 48G memory consumption.
When ram_cache_cutoff is set to 2MB, the pictures are not be cached into RAM. ATS will not crash.
Tt seems that memory leaking happens in ramcache, or it is a bug of jemalloc-5.2.1?

The crash log as follows:
Process: [TS_MAIN] [33703]
Version: Traffic Server 9.1.4
System Version: Linux x86_64 #1 SMP Sat Mar 14 01:26:23 CST 2020 4.18.0-147.3.1.el7_1.x86_64
Date: Fri, 07 Jul 2023 14:23:33 +0800

Signal Status:
siginfo.si_signo: 6 (Aborted)
siginfo.si_pid: 33703 siginfo.si_uid: 12352
siginfo.si_code: 0xfffffffa (-6) siginfo.si_addr: 0x00003040000083a7

CPU Registers:
R8 :0x0000000000000088 R9 :0x00007fe6c066a018 R10:0x0000000000000008 R11:0x0000000000000202
R12:0x0000000000001000 R13:0x0000000000000000 R14:0x00007fe6b1e9a050 R15:0x0000000000100000
RDI:0x00000000000083a7 RSI:0x000000000000842b RBP:0x0000000000100000 RBX:0x0000000000001000
RDX:0x0000000000000006 RAX:0x0000000000000000 RCX:0x00007fe6bf205487 RSP:0x00007fe697a0beb8
RIP:0x00007fe6bf205487 EFL:0x0000000000000202 CSGSFS:0x002b000000000033 ERR:0x0000000000000000
TRAPNO:0x0000000000000000 OLDMASK:0x0000000000000000 CR2:0x0000000000000000

Unable to retrieve backtrace: [6] Error reading from socket.

Process Status:
Name: [TS_MAIN]
Umask: 0027
State: S (sleeping)
Tgid: 33703
Ngid: 0
Pid: 33703
PPid: 33693
TracerPid: 0
Uid: 12352 12352 12352 12352
Gid: 12352 12352 12352 12352
FDSize: 32768
Groups: 6 12352
NStgid: 33703
NSpid: 33703
NSpgid: 33692
NSsid: 1330
VmPeak: 44411796 kB
VmSize: 44411796 kB
VmLck: 0 kB
VmPin: 0 kB
VmHWM: 36940536 kB
VmRSS: 36940536 kB
RssAnon: 36928164 kB
RssFile: 12372 kB
RssShmem: 0 kB
VmData: 44328600 kB
VmStk: 132 kB
VmExe: 3900 kB
VmLib: 10416 kB
VmPTE: 79432 kB
VmSwap: 0 kB
HugetlbPages: 0 kB
CoreDumping: 0
Threads: 376
SigQ: 0/513058
SigPnd: 0000000000000000
ShdPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000001001
SigCgt: 0000000180004efe
CapInh: 0000000000000000
CapPrm: 0000000000000400
CapEff: 0000000000000400
CapBnd: 0000003fffffffff
CapAmb: 0000000000000000
NoNewPrivs: 0
Seccomp: 0
Speculation_Store_Bypass: vulnerable
Cpus_allowed: ff,ffffffff
Cpus_allowed_list: 0-39
Mems_allowed: 00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000003
Mems_allowed_list: 0-1
voluntary_ctxt_switches: 885
nonvoluntary_ctxt_switches: 43

Process Limits:
Limit Soft Limit Hard Limit Units
Max cpu time unlimited unlimited seconds
Max file size unlimited unlimited bytes
Max data size unlimited unlimited bytes
Max stack size 4194304 4194304 bytes
Max core file size 0 0 bytes
Max resident set unlimited unlimited bytes
Max processes 4096 513058 processes
Max open files 65535 65535 files
Max locked memory 65536 65536 bytes
Max address space unlimited unlimited bytes
Max file locks unlimited unlimited locks
Max pending signals 513058 513058 signals
Max msgqueue size 819200 819200 bytes
Max nice priority 0 0
Max realtime priority 0 0
Max realtime timeout unlimited unlimited us

Unable to retrieve Traffic Server records: [6] Error reading from socket.

Metadata

Metadata

Assignees

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions