runtime: GC & debug.FreeOSMemory calls don't return memory to the OS

[AlexBlack772]

What version of Go are you using (go version)?

$ go version
go version go1.15.5 linux/amd64
CentOS Linux release 7.7.1908 (Core)
kernel-3.10.0-1062.el7.x86_64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="direct"
GOROOT="/usr/lib/golang"
GOSUMDB="off"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/golang/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/var/www/virtual/service.extractor/service/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build181628151=/tmp/go-build -gno-record-gcc-switches"

What did you do?

A programm allocates some memory and than release it.

https://play.golang.org/p/Ljr0c6zRasB

What did you expect to see?

The programm releases allocated memor and after runtime.GC() & debug.FreeOSMemory() calls I expect that values of VmSize/VmData drop, but nothing happen.

What did you see instead?

Values of VmSize/VmData remains the same

-- (comment) the programm started

START VmSize:725Mb VmLck:0 VmPin:0 VmRSS:1 VmData:717 VmStk:0 VmExe:0 VmLib:1

-- (comment) the programm allocated memory for buffers

  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:21  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1

END
END
....
END

-- (comment) the programm released memory and shoud return it to the system

  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1
  VmSize:1321Mb  VmLck:0  VmPin:0  VmRSS:22  VmData:1313  VmStk:0  VmExe:0  VmLib:1

I tried to use "env GODEBUG=madvdontneed=1 ./test2" but it didn't help, it seems that kernel 3.10.0 always use MADV_DONTNEED and GODEBUG=madvdontneed=1 changes nothing

pmap also saw allocated memory

[root@localhost go]# pmap -x  5008
5008:   ./test2
Address           Kbytes     RSS   Dirty Mode  Mapping
0000000000400000     828     672       0 r-x-- test2
00000000004cf000     776     396       0 r---- test2
0000000000591000     100      68      40 rw--- test2
00000000005aa000     204      64      64 rw---   [ anon ]
0000000002089000     132       4       4 rw---   [ anon ]
000000c000000000  655360     480     480 rw---   [ anon ]
00007f3e3a1d8000   20964   18100   18100 rw---   [ anon ]
00007f3e3b651000       4       0       0 -----   [ anon ]
00007f3e3b652000    8192       8       8 rw---   [ anon ]
00007f3e3be52000       4       0       0 -----   [ anon ]
00007f3e3be53000    8192       8       8 rw---   [ anon ]
00007f3e3c653000       4       0       0 -----   [ anon ]
00007f3e3c654000    8192       8       8 rw---   [ anon ]
00007f3e3ce54000       4       0       0 -----   [ anon ]
00007f3e3ce55000   44100    2148    2148 rw---   [ anon ]
00007f3e3f966000  263680       0       0 -----   [ anon ]
00007f3e4fae6000       4       4       4 rw---   [ anon ]
00007f3e4fae7000  293564       0       0 -----   [ anon ]
00007f3e61996000       4       4       4 rw---   [ anon ]
00007f3e61997000   36692       0       0 -----   [ anon ]
00007f3e63d6c000       4       4       4 rw---   [ anon ]
00007f3e63d6d000    4068       0       0 -----   [ anon ]
00007f3e64166000    1804     264       0 r-x-- libc-2.17.so
00007f3e64329000    2048       0       0 ----- libc-2.17.so
00007f3e64529000      16      16      16 r---- libc-2.17.so
00007f3e6452d000       8       8       8 rw--- libc-2.17.so
00007f3e6452f000      20      12      12 rw---   [ anon ]
00007f3e64534000      92      56       0 r-x-- libpthread-2.17.so
00007f3e6454b000    2044       0       0 ----- libpthread-2.17.so
00007f3e6474a000       4       4       4 r---- libpthread-2.17.so
00007f3e6474b000       4       4       4 rw--- libpthread-2.17.so
00007f3e6474c000      16       4       4 rw---   [ anon ]
00007f3e64750000     136     108       0 r-x-- ld-2.17.so
00007f3e6477f000     512     416     416 rw---   [ anon ]
00007f3e647ff000     512       0       0 -----   [ anon ]
00007f3e6487f000       4       4       4 rw---   [ anon ]
00007f3e64880000     508       0       0 -----   [ anon ]
00007f3e648ff000     396      44      44 rw---   [ anon ]
00007f3e64970000       4       4       4 rw---   [ anon ]
00007f3e64971000       4       4       4 r---- ld-2.17.so
00007f3e64972000       4       4       4 rw--- ld-2.17.so
00007f3e64973000       4       4       4 rw---   [ anon ]
00007ffd8d92f000     132      20      20 rw---   [ stack ]
00007ffd8d9c6000       8       4       0 r-x--   [ anon ]
ffffffffff600000       4       0       0 r-x--   [ anon ]
---------------- ------- ------- -------
total kB         1353356   22948   21420

[cherrymui]

cc @aclements @mknyszek

[mknyszek]

@AlexBlack772 VmSize is virtual memory footprint and for a Go application it always grows monotonically; the Go runtime never unmaps heap memory. It appears that VmData is just VmSize with a few things subtracted out (https://ewx.livejournal.com/579283.html), so I expect it to have ~the same effect. In generally, address space is cheap on modern systems and does not reflect actual physical memory resources used. When we return memory to the OS, we mark that memory as "unused" to the OS and it's up to the OS to take that memory back. The result is that VmRSS goes down.

VmRSS is the actual physical memory footprint, and Linux won't actually take that memory back until your system is under memory pressure in Go 1.15, because we use MADV_FREE by default (as you point out, though, Linux 3.10.0 doesn't have MADV_FREE, so we fall back on MADV_DONTNEED anyway). In Go 1.16 and above, GODEBUG=madvdontneed=1 is the default.

Tentatively closing this issue as WAI. Please reopen if you feel this is in error.