all: resident set size keeps on increasing, but no memory leaks

[cupen]

What version of Go are you using (go version)?

$ go version
go version go1.14.1 linux/amd64

Does this issue reproduce with the latest release?

yes, it's also reproduce with go1.13.7

What operating system and processor architecture are you using (go env)?

• archlinux (compile)
• centos7.4 + kernel3.10 (runtime)

go env Output

$ go env
GO111MODULE="off"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/cupen/.cache/go-build"
GOENV="/home/cupen/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOOS="linux"
GOPATH="/home/cupen/go"
GOPROXY="https://goproxy.cn"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build030297143=/tmp/go-build -gno-record-gcc-switches"

What did you do?

My http service meet a memory problem. The heap spikes causing RSS increasing, but after heap spike the RSS doesn't decreasing. There is no memory leaks, it's checked via pprof and monitor(prometheus).

What did you expect to see?

After heap spikes, RSS will be decreasing.

What did you see instead?

After heap spikes RSS keeps on increasing.

pmap -x 15080 Output

15080:   /data/the/path/of/my/server web
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000   14560    7944       0 r-x-- server (deleted)
0000000001238000   18212   10768       0 r---- server (deleted)
0000000002401000     432     272     144 rw--- server (deleted)
000000000246d000     224     104     104 rw---   [ anon ]
0000000003274000     132       4       4 rw---   [ anon ]
000000c000000000 12779520 9599484 9599484 rw---   [ anon ]
00007fa44bae7000  132640  129236  129236 rw---   [ anon ]
00007fa453c6f000       4       0       0 -----   [ anon ]
00007fa453c70000  182000  172868  172868 rw---   [ anon ]
00007fa45ee34000   19544   19436   19436 rw---   [ anon ]
00007fa46014a000       4       0       0 -----   [ anon ]
00007fa46014b000   51640   43216   43216 rw---   [ anon ]
00007fa4633c4000   52876   52712   52712 rw---   [ anon ]
00007fa466769000   16984   16932   16932 rw---   [ anon ]
00007fa4677ff000       4       0       0 -----   [ anon ]
00007fa467800000    8192       8       8 rw---   [ anon ]
00007fa468000000     132       4       4 rw---   [ anon ]
00007fa468021000   65404       0       0 -----   [ anon ]
00007fa46c000000     132       4       4 rw---   [ anon ]
00007fa46c021000   65404       0       0 -----   [ anon ]
00007fa470009000   15828   15736   15736 rw---   [ anon ]
00007fa470f85000   18328   18284   18284 rw---   [ anon ]
00007fa472175000   12944   12884   12884 rw---   [ anon ]
00007fa472e1a000   18328   18304   18304 rw---   [ anon ]
00007fa474000000     132       4       4 rw---   [ anon ]
00007fa474021000   65404       0       0 -----   [ anon ]
00007fa47800b000    7240    7224    7224 rw---   [ anon ]
00007fa478723000    8908    8904    8904 rw---   [ anon ]
00007fa478fda000     128     128     128 rw---   [ anon ]
00007fa478ffa000       4       0       0 -----   [ anon ]
00007fa478ffb000    8192       8       8 rw---   [ anon ]
00007fa4797fb000       4       0       0 -----   [ anon ]
00007fa4797fc000    8192       8       8 rw---   [ anon ]
00007fa479ffc000       4       0       0 -----   [ anon ]
00007fa479ffd000    8192       8       8 rw---   [ anon ]
00007fa47a7fd000       4       0       0 -----   [ anon ]
00007fa47a7fe000    8192       8       8 rw---   [ anon ]
00007fa47affe000       4       0       0 -----   [ anon ]
00007fa47afff000    8192       8       8 rw---   [ anon ]
00007fa47b7ff000       4       0       0 -----   [ anon ]
00007fa47b800000    8192    2048    2048 rw---   [ anon ]
00007fa47c000000     132       4       4 rw---   [ anon ]
00007fa47c021000   65404       0       0 -----   [ anon ]
00007fa480000000     132       4       4 rw---   [ anon ]
00007fa480021000   65404       0       0 -----   [ anon ]
00007fa484000000     132       4       4 rw---   [ anon ]
00007fa484021000   65404       0       0 -----   [ anon ]
00007fa488004000    1408    1388    1388 rw---   [ anon ]
00007fa48816c000    5896    5896    5896 rw---   [ anon ]
00007fa48872e000       4       0       0 -----   [ anon ]
00007fa48872f000    9856    1672    1672 rw---   [ anon ]
00007fa4890cf000       4       0       0 -----   [ anon ]
00007fa4890d0000    8704     516     516 rw---   [ anon ]
00007fa489950000       4       0       0 -----   [ anon ]
00007fa489951000    8192       8       8 rw---   [ anon ]
00007fa48a151000       4       0       0 -----   [ anon ]
00007fa48a152000    8192       8       8 rw---   [ anon ]
00007fa48a952000       4       0       0 -----   [ anon ]
00007fa48a953000    8192       8       8 rw---   [ anon ]
00007fa48b153000       4       0       0 -----   [ anon ]
00007fa48b154000    8192       8       8 rw---   [ anon ]
00007fa48b954000       4       0       0 -----   [ anon ]
00007fa48b955000   44100    4904    4904 rw---   [ anon ]
00007fa48e466000  263680       0       0 -----   [ anon ]
00007fa49e5e6000      28      28      28 rw---   [ anon ]
00007fa49e5ed000  293540       0       0 -----   [ anon ]
00007fa4b0496000       4       4       4 rw---   [ anon ]
00007fa4b0497000   36692       0       0 -----   [ anon ]
00007fa4b286c000       4       4       4 rw---   [ anon ]
00007fa4b286d000    4068       0       0 -----   [ anon ]
00007fa4b2c66000    1804     248       0 r-x-- libc-2.17.so
00007fa4b2e29000    2048       0       0 ----- libc-2.17.so
00007fa4b3029000      16      16      16 r---- libc-2.17.so
00007fa4b302d000       8       8       8 rw--- libc-2.17.so
00007fa4b302f000      20      12      12 rw---   [ anon ]
00007fa4b3034000      92      56       0 r-x-- libpthread-2.17.so
00007fa4b304b000    2044       0       0 ----- libpthread-2.17.so
00007fa4b324a000       4       4       4 r---- libpthread-2.17.so
00007fa4b324b000       4       4       4 rw--- libpthread-2.17.so
00007fa4b324c000      16       4       4 rw---   [ anon ]
00007fa4b3250000     136     108       0 r-x-- ld-2.17.so
00007fa4b3277000     576     576     576 rw---   [ anon ]
00007fa4b3307000     512       0       0 -----   [ anon ]
00007fa4b3387000       4       4       4 rw---   [ anon ]
00007fa4b3388000     508       0       0 -----   [ anon ]
00007fa4b3407000     396      68      68 rw---   [ anon ]
00007fa4b3470000       4       4       4 rw---   [ anon ]
00007fa4b3471000       4       4       4 r---- ld-2.17.so
00007fa4b3472000       4       4       4 rw--- ld-2.17.so
00007fa4b3473000       4       4       4 rw---   [ anon ]
00007ffc90470000     132      20      20 rw---   [ stack ]
00007ffc904f0000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]

cat /proc/15080/status Output

Name:	server
Umask:	0022
State:	R (running)
Tgid:	15080
Ngid:	0
Pid:	15080
PPid:	13916
TracerPid:	0
Uid:	1000	1000	1000	1000
Gid:	100	100	100	100
FDSize:	32768
Groups:	100 1000 
VmPeak:	14510184 kB
VmSize:	14510184 kB
VmLck:	       0 kB
VmPin:	       0 kB
VmHWM:	13230816 kB
VmRSS:	10002652 kB
RssAnon:	 9983548 kB
RssFile:	   19104 kB
RssShmem:	       0 kB
VmData:	14470684 kB
VmStk:	     132 kB
VmExe:	   14560 kB
VmLib:	    2032 kB
VmPTE:	   26124 kB
VmSwap:	       0 kB
Threads:	17
SigQ:	0/63456
SigPnd:	0000000000000000
ShdPnd:	0000000000000000
SigBlk:	0000000000000000
SigIgn:	0000000000000000
SigCgt:	ffffffffffc1feff
CapInh:	0000000000000000
CapPrm:	0000000000000000
CapEff:	0000000000000000
CapBnd:	0000001fffffffff
CapAmb:	0000000000000000
NoNewPrivs:	0
Seccomp:	0
Speculation_Store_Bypass:	vulnerable
Cpus_allowed:	ff
Cpus_allowed_list:	0-7
Mems_allowed:	00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000000,00000001
Mems_allowed_list:	0
voluntary_ctxt_switches:	46090939
nonvoluntary_ctxt_switches:	1154004

[cupen]

It seems like the alloc speed is lager than release? the RSS = heap_inuse + heap_alloc ?

[andybons]

Thanks for the issue.

This is expected and RSS will drop if the system is under memory pressure.

From https://golang.org/doc/go1.12:

On Linux, the runtime now uses MADV_FREE to release unused memory. This is more efficient but may result in higher reported RSS. The kernel will reclaim the unused data when it is needed.

More info here: #29337 (comment)

@aclements @mknyszek