Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cgroup.InContainer is wrong in some case #38659

Closed
lance6716 opened this issue Oct 26, 2022 · 5 comments
Closed

cgroup.InContainer is wrong in some case #38659

lance6716 opened this issue Oct 26, 2022 · 5 comments
Assignees
Labels
feature/developing the related feature is in development severity/minor sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.

Comments

@lance6716
Copy link
Contributor

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. Minimal reproduce step (Required)

func InContainer() bool {
	v, err := os.ReadFile(procPathCGroup)
	if err != nil {
		return false
	}
	if strings.Contains(string(v), "docker") ||
		strings.Contains(string(v), "kubepods") ||
		strings.Contains(string(v), "containerd") {
		return true
	}
	return false
}

where

procPathCGroup    = "/proc/self/cgroup"

then

$ docker run -it --rm --memory=300M golang:1.19-alpine sh 
/go # cat /proc/self/cgroup
0::/
/go # 

2. What did you expect to see? (Required)

we know it's inside a container

3. What did you see instead (Required)

4. What is your TiDB version? (Required)

@lance6716 lance6716 added the type/bug The issue is confirmed as a bug. label Oct 26, 2022
@lance6716
Copy link
Contributor Author

however, as @amyangfei tested, the function can correctly retrieve the memory limit

img_v2_f759e3b0-bbb6-4834-a680-71bfd9cbd56g

/assign @hawkingrei

@lance6716
Copy link
Contributor Author

according to "/proc files" section in https://man7.org/linux/man-pages/man7/cgroups.7.html#NOTES

                 For cgroups version 1 hierarchies, this field contains
                 a comma-separated list of the controllers bound to the
                 hierarchy.  For the cgroups version 2 hierarchy, this
                 field is empty.

we should find another way to check cgroup for version 2

@ti-chi-bot ti-chi-bot added may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.0 may-affects-6.1 may-affects-6.2 may-affects-6.3 labels Oct 27, 2022
@VelocityLight VelocityLight removed may-affects-5.0 This bug maybe affects 5.0.x versions. may-affects-5.1 This bug maybe affects 5.1.x versions. may-affects-5.2 This bug maybe affects 5.2.x versions. may-affects-5.3 This bug maybe affects 5.3.x versions. may-affects-5.4 This bug maybe affects 5.4.x versions. may-affects-6.1 may-affects-6.2 may-affects-6.3 labels Oct 27, 2022
@xhebox xhebox added severity/minor and removed severity/major may-affects-4.0 This bug maybe affects 4.0.x versions. may-affects-6.0 labels Oct 27, 2022
@xhebox
Copy link
Contributor

xhebox commented Oct 27, 2022

It is not even in release-6.3. Function is added last month. Even if it is a bug, this should not be a major issue.

@xhebox xhebox added the feature/developing the related feature is in development label Oct 27, 2022
@lilinghai
Copy link
Contributor

Maybe we can statfs the /proc/self/cgroup to check the file type according to the

CGROUP_SUPER_MAGIC    0x27e0eb   /* Cgroup pseudo FS */
CGROUP2_SUPER_MAGIC   0x63677270 /* Cgroup v2 pseudo FS */

https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html#mounting ,
and then parse the memory controller in the cgroup2 type mount point /sys/fs/cgroup (check the mount point using mount -t cgourp2 command).

@Defined2014
Copy link
Contributor

Defined2014 commented Nov 13, 2023

Similarly, cgroup.InContainer will also incorrectly determine that it is running in a container. I run TestCPUValue in non-container env and failed. I run cat /proc/self/mountinfo get the info below

22 27 0:20 / /sys rw,nosuid,nodev,noexec,relatime shared:7 - sysfs sysfs rw
23 27 0:21 / /proc rw,nosuid,nodev,noexec,relatime shared:13 - proc proc rw
24 27 0:5 / /dev rw,nosuid,relatime shared:2 - devtmpfs udev rw,size=14295464k,nr_inodes=3573866,mode=755
25 24 0:22 / /dev/pts rw,nosuid,noexec,relatime shared:3 - devpts devpts rw,gid=5,mode=620,ptmxmode=000
26 27 0:23 / /run rw,nosuid,nodev,noexec,relatime shared:5 - tmpfs tmpfs rw,size=2870412k,mode=755
27 1 259:2 / / rw,relatime shared:1 - ext4 /dev/nvme0n1p2 rw,errors=remount-ro
28 22 0:6 / /sys/kernel/security rw,nosuid,nodev,noexec,relatime shared:8 - securityfs securityfs rw
29 24 0:24 / /dev/shm rw,nosuid,nodev shared:4 - tmpfs tmpfs rw
30 26 0:25 / /run/lock rw,nosuid,nodev,noexec,relatime shared:6 - tmpfs tmpfs rw,size=5120k
31 22 0:26 / /sys/fs/cgroup rw,nosuid,nodev,noexec,relatime shared:9 - cgroup2 cgroup2 rw,nsdelegate,memory_recursiveprot
32 22 0:27 / /sys/fs/pstore rw,nosuid,nodev,noexec,relatime shared:10 - pstore pstore rw
33 22 0:28 / /sys/firmware/efi/efivars rw,nosuid,nodev,noexec,relatime shared:11 - efivarfs efivarfs rw
34 22 0:29 / /sys/fs/bpf rw,nosuid,nodev,noexec,relatime shared:12 - bpf none rw,mode=700
35 23 0:30 / /proc/sys/fs/binfmt_misc rw,relatime shared:14 - autofs systemd-1 rw,fd=29,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=15385
36 24 0:19 / /dev/mqueue rw,nosuid,nodev,noexec,relatime shared:15 - mqueue mqueue rw
37 22 0:7 / /sys/kernel/debug rw,nosuid,nodev,noexec,relatime shared:16 - debugfs debugfs rw
38 24 0:31 / /dev/hugepages rw,relatime shared:17 - hugetlbfs hugetlbfs rw,pagesize=2M
39 22 0:11 / /sys/kernel/tracing rw,nosuid,nodev,noexec,relatime shared:18 - tracefs tracefs rw
40 22 0:32 / /sys/fs/fuse/connections rw,nosuid,nodev,noexec,relatime shared:19 - fusectl fusectl rw
41 22 0:33 / /sys/kernel/config rw,nosuid,nodev,noexec,relatime shared:20 - configfs configfs rw
42 35 0:34 / /proc/sys/fs/binfmt_misc rw,nosuid,nodev,noexec,relatime shared:21 - binfmt_misc binfmt_misc rw
90 27 259:1 / /boot/efi rw,relatime shared:46 - vfat /dev/nvme0n1p1 rw,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=mixed,utf8,errors=remount-ro
382 27 0:45 / /var/lib/docker/overlay2/d4b43436f633cb3eb50f5a27e79cf55340fef2f000ad6f75a8bf2ff5920e20fc/merged rw,relatime shared:186 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/G4GQRDA
U73PKJUVNEHNJ5APQ5L:/var/lib/docker/overlay2/l/SUVTSPGY5QQ3OHTSRIC6STI5N6:/var/lib/docker/overlay2/l/M4KYJP3ANDCSQOCN5XSNN3KHNX:/var/lib/docker/overlay2/l/3HKQDERX3UY3ZXGH55CRS324TG:/var/lib/docker/
overlay2/l/SX75CK6T2F2OYB7Z7SN3LJ5KW2:/var/lib/docker/overlay2/l/BLT46MJK6IMMOIEGCXD6T3IW4W:/var/lib/docker/overlay2/l/H2IRR7PUNW3SLJCLRJNLWTTFX3,upperdir=/var/lib/docker/overlay2/d4b43436f633cb3eb5
0f5a27e79cf55340fef2f000ad6f75a8bf2ff5920e20fc/diff,workdir=/var/lib/docker/overlay2/d4b43436f633cb3eb50f5a27e79cf55340fef2f000ad6f75a8bf2ff5920e20fc/work
403 27 0:42 / /var/lib/docker/overlay2/5360a4127a7fcefdaf06d146d2913b6b436cee1830d96e32f236c1162ba8e2be/merged rw,relatime shared:193 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/J2SNK5W
5VWYWQ2MS3TFH3SBLVH:/var/lib/docker/overlay2/l/I3G666N5T5L62SXTORQJK4FMRE:/var/lib/docker/overlay2/l/4BQ7VXK7IKTQ4WV4HT6AG2OTQ6:/var/lib/docker/overlay2/l/RNKJ6ZR3MTZ4YVKKQCN6PHQVTJ:/var/lib/docker/
overlay2/l/7DII4G3YDBVZISJVUPAQP5A7R5:/var/lib/docker/overlay2/l/GAPSB3UKAQG2TWJVB6FADX2S5B,upperdir=/var/lib/docker/overlay2/5360a4127a7fcefdaf06d146d2913b6b436cee1830d96e32f236c1162ba8e2be/diff,wo
rkdir=/var/lib/docker/overlay2/5360a4127a7fcefdaf06d146d2913b6b436cee1830d96e32f236c1162ba8e2be/work
430 27 0:46 / /var/lib/docker/overlay2/1afb95336fb5c74a85a9336e7be249ddf70f9ab90b1eac208557456017638785/merged rw,relatime shared:200 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/D4KM5AY
ALJ5K3AA7DSG7T6M2GM:/var/lib/docker/overlay2/l/PO5BB66QL5QNQEDSQXI2YGVVVW:/var/lib/docker/overlay2/l/3HQDJAJL4BFNKAOJAAYR2BMHE6:/var/lib/docker/overlay2/l/OONAZJNR6ODJ2XQRLRV2CVGQM4:/var/lib/docker/
overlay2/l/KCUYAKZNDRPM4SKTN4P3GOGTZW:/var/lib/docker/overlay2/l/HMRBL7MVCDVAYBO7WIUJYXVWKX:/var/lib/docker/overlay2/l/3CCP2D5WXFCFXKREE47KNALMYV:/var/lib/docker/overlay2/l/VUIT5Q3WMSHZEDSZVZRWB4GIX
O:/var/lib/docker/overlay2/l/H2IRR7PUNW3SLJCLRJNLWTTFX3,upperdir=/var/lib/docker/overlay2/1afb95336fb5c74a85a9336e7be249ddf70f9ab90b1eac208557456017638785/diff,workdir=/var/lib/docker/overlay2/1afb9
5336fb5c74a85a9336e7be249ddf70f9ab90b1eac208557456017638785/work
466 27 0:39 / /var/lib/docker/overlay2/ffa0f7c3e1ae5aed9c34168d714f0d5d61deb6cb99e009f9af07a8a2ac4b42ae/merged rw,relatime shared:207 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/RMLSWTC
HVXHGJ45T5UXQQENPX4:/var/lib/docker/overlay2/l/UILIFUQZTFXDDXAFDBNH25E6ZP:/var/lib/docker/overlay2/l/C2HKVFZ7CJLZOYSDB6MSAH2KVI:/var/lib/docker/overlay2/l/2QMXNXNJWZ4FPO2SOV6THVM476:/var/lib/docker/
overlay2/l/OTTZ6HMG3ZYP5AXUNIYOBEU3QQ:/var/lib/docker/overlay2/l/CVB3VJTZ6ZGX2R5BWB4OW5FCW6:/var/lib/docker/overlay2/l/SX2FNZNX5ZML544NJITFILLDOP:/var/lib/docker/overlay2/l/FR2WLCRXME6SLRY3GOLG323F4
G:/var/lib/docker/overlay2/l/VEFUERKBCLA4HULM3BE42LARFT:/var/lib/docker/overlay2/l/T7XWUGHB4ZKTN52SLV7YCDG4NI:/var/lib/docker/overlay2/l/Q6BGAXCWSR5U4C7NNI7NHZIGLD:/var/lib/docker/overlay2/l/H2IRR7P
UNW3SLJCLRJNLWTTFX3,upperdir=/var/lib/docker/overlay2/ffa0f7c3e1ae5aed9c34168d714f0d5d61deb6cb99e009f9af07a8a2ac4b42ae/diff,workdir=/var/lib/docker/overlay2/ffa0f7c3e1ae5aed9c34168d714f0d5d61deb6cb9
9e009f9af07a8a2ac4b42ae/work
490 27 0:40 / /var/lib/docker/overlay2/03547e50f821913ed5319865249f1e9ae418234ac14f21bb8f0d9a3df24a18f8/merged rw,relatime shared:214 - overlay overlay rw,lowerdir=/var/lib/docker/overlay2/l/VARYU6K
67CIRZ7SOXOMIAZMS5L:/var/lib/docker/overlay2/l/WZYNDRVDZC66RHUAZCDX732IQF:/var/lib/docker/overlay2/l/5T5RRJADBKZPNAAKXR73SZ5VYA:/var/lib/docker/overlay2/l/BITRXIM5C3MG3ECVQ4YIGC7EVK:/var/lib/docker/
overlay2/l/ASKFCF4M4L4KNGM55O2AGB4WAS:/var/lib/docker/overlay2/l/GYHMB6OEY6AV2FNFSPS5NLMBLS:/var/lib/docker/overlay2/l/DJRW6FZ2ONZMHRBC2HJUW256P5:/var/lib/docker/overlay2/l/VYZB7KZXWAZMX5EOU3XWNXRSO
C:/var/lib/docker/overlay2/l/2CLAGPIGVJESUMX4HE6E2IFOUM:/var/lib/docker/overlay2/l/V2UWMX67WW6AR33ZPSYOFFORCJ:/var/lib/docker/overlay2/l/ELY6UXNZ43O3D5GMO4PCPRDZYV,upperdir=/var/lib/docker/overlay2/
03547e50f821913ed5319865249f1e9ae418234ac14f21bb8f0d9a3df24a18f8/diff,workdir=/var/lib/docker/overlay2/03547e50f821913ed5319865249f1e9ae418234ac14f21bb8f0d9a3df24a18f8/work
...

I'm not an expert for docker, but I think this just means we have a docker env running on this OS, not that we're running this test in a container.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature/developing the related feature is in development severity/minor sig/sql-infra SIG: SQL Infra type/bug The issue is confirmed as a bug.
Projects
None yet
Development

No branches or pull requests

8 participants