Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

podman stats crash with network mode host #5652

Closed
nileshgr opened this issue Mar 29, 2020 · 3 comments · Fixed by #6470
Closed

podman stats crash with network mode host #5652

nileshgr opened this issue Mar 29, 2020 · 3 comments · Fixed by #6470
Assignees
@mheon
Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue

Comments

@nileshgr
Copy link

Is this a BUG REPORT or FEATURE REQUEST? (leave only one on its own line)

/kind bug

Description

podman panics / crashes with segfault when one pod is running and podman pod stats or podman stats is run when pod is in host network mode

Steps to reproduce the issue:

  1. podman pod create --network host -n test

  2. podman run -it --rm --pod test centos:8 /bin/bash

  3. podman pod stats test

Describe the results you received:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x556e4f00cd6c]

goroutine 1 [running]:
panic(0x556e4fce06e0, 0x556e51231330)
	/usr/lib/golang/src/runtime/panic.go:565 +0x2c9 fp=0xc0004bb8a0 sp=0xc0004bb810 pc=0x556e4df31b79
runtime.panicmem(...)
	/usr/lib/golang/src/runtime/panic.go:82
runtime.sigpanic()
	/usr/lib/golang/src/runtime/signal_unix.go:390 +0x415 fp=0xc0004bb8d0 sp=0xc0004bb8a0 pc=0x556e4df477a5
github.com/containers/libpod/libpod.getContainerNetNS(0xc000276280, 0xc0001f4fa0, 0x0, 0x0, 0x0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/libpod/networking_linux.go:526 +0xdc fp=0xc0004bb918 sp=0xc0004bb8d0 pc=0x556e4f00cd6c
github.com/containers/libpod/libpod.getContainerNetIO(0xc000276280, 0xc000214b40, 0x0, 0x0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/libpod/networking_linux.go:539 +0x5a fp=0xc0004bb968 sp=0xc0004bb918 pc=0x556e4f00ce6a
github.com/containers/libpod/libpod.(*Container).GetContainerStats(0xc000276280, 0xc000276460, 0x0, 0x0, 0x0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/libpod/stats.go:52 +0x173 fp=0xc0004bba20 sp=0xc0004bb968 pc=0x556e4f046d73
github.com/containers/libpod/libpod.(*Pod).GetPodStats(0xc000272a80, 0xc0002955c0, 0x0, 0x0, 0x0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/libpod/pod.go:295 +0x1b5 fp=0xc0004bbaa8 sp=0xc0004bba20 pc=0x556e4f0265d5
main.podStatsCmd(0x556e512c0ac0, 0x0, 0x0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/cmd/podman/pod_stats.go:140 +0x46c fp=0xc0004bbd68 sp=0xc0004bbaa8 pc=0x556e4f25b3bc
main.glob..func91(0x556e51251c80, 0xc00042aaf0, 0x1, 0x1, 0x0, 0x0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/cmd/podman/pod_stats.go:36 +0x88 fp=0xc0004bbd90 sp=0xc0004bbd68 pc=0x556e4f28c608
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).execute(0x556e51251c80, 0xc00003c0f0, 0x1, 0x1, 0x556e51251c80, 0xc00003c0f0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:840 +0x467 fp=0xc0004bbe78 sp=0xc0004bbd90 pc=0x556e4e0f0cd7
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x556e5124efe0, 0x6, 0x556e4f2a73dd, 0x8)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:945 +0x31a fp=0xc0004bbf50 sp=0xc0004bbe78 pc=0x556e4e0f183a
github.com/containers/libpod/vendor/github.com/spf13/cobra.(*Command).Execute(...)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/github.com/spf13/cobra/command.go:885
main.main()
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/cmd/podman/main.go:162 +0xb2 fp=0xc0004bbf98 sp=0xc0004bbf50 pc=0x556e4f24de22
runtime.main()
	/usr/lib/golang/src/runtime/proc.go:200 +0x214 fp=0xc0004bbfe0 sp=0xc0004bbf98 pc=0x556e4df338d4
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc0004bbfe8 sp=0xc0004bbfe0 pc=0x556e4df5fbe1

goroutine 2 [force gc (idle)]:
runtime.gopark(0x556e4ff43d70, 0x556e512b4e00, 0x1410, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00005efb0 sp=0xc00005ef90 pc=0x556e4df33cd5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.forcegchelper()
	/usr/lib/golang/src/runtime/proc.go:250 +0xbb fp=0xc00005efe0 sp=0xc00005efb0 pc=0x556e4df33b6b
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005efe8 sp=0xc00005efe0 pc=0x556e4df5fbe1
created by runtime.init.6
	/usr/lib/golang/src/runtime/proc.go:239 +0x37

goroutine 3 [GC sweep wait]:
runtime.gopark(0x556e4ff43d70, 0x556e512b71e0, 0x140c, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00005f7a8 sp=0xc00005f788 pc=0x556e4df33cd5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.bgsweep(0xc00007e000)
	/usr/lib/golang/src/runtime/mgcsweep.go:89 +0x138 fp=0xc00005f7d8 sp=0xc00005f7a8 pc=0x556e4df26978
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005f7e0 sp=0xc00005f7d8 pc=0x556e4df5fbe1
created by runtime.gcenable
	/usr/lib/golang/src/runtime/mgc.go:208 +0x5a

goroutine 4 [finalizer wait]:
runtime.gopark(0x556e4ff43d70, 0x556e512e4038, 0x140f, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00005ff58 sp=0xc00005ff38 pc=0x556e4df33cd5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.runfinq()
	/usr/lib/golang/src/runtime/mfinal.go:175 +0xaa fp=0xc00005ffe0 sp=0xc00005ff58 pc=0x556e4df1d4da
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005ffe8 sp=0xc00005ffe0 pc=0x556e4df5fbe1
created by runtime.createfing
	/usr/lib/golang/src/runtime/mfinal.go:156 +0x63

goroutine 5 [syscall]:
runtime.notetsleepg(0x556e512cae00, 0x12a05ebaa, 0x556e4ec11414)
	/usr/lib/golang/src/runtime/lock_futex.go:227 +0x38 fp=0xc00005e760 sp=0xc00005e730 pc=0x556e4df0fad8
runtime.timerproc(0x556e512cade0)
	/usr/lib/golang/src/runtime/time.go:311 +0x2ee fp=0xc00005e7d8 sp=0xc00005e760 pc=0x556e4df50f1e
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005e7e0 sp=0xc00005e7d8 pc=0x556e4df5fbe1
created by runtime.(*timersBucket).addtimerLocked
	/usr/lib/golang/src/runtime/time.go:169 +0x110

goroutine 40 [select]:
runtime.gopark(0x556e4ff43db0, 0x0, 0x1809, 0x1)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc000061d50 sp=0xc000061d30 pc=0x556e4df33cd5
runtime.selectgo(0xc000061f48, 0xc000061ec4, 0x3, 0x556e4df5c650, 0xc000000180)
	/usr/lib/golang/src/runtime/select.go:313 +0xcbe fp=0xc000061e78 sp=0xc000061d50 pc=0x556e4df4386e
github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.(*cniNetworkPlugin).monitorConfDir(0xc000104d80, 0xc000374370)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:152 +0x19a fp=0xc000061fd0 sp=0xc000061e78 pc=0x556e4eea5e1a
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc000061fd8 sp=0xc000061fd0 pc=0x556e4df5fbe1
created by github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni.initCNI
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/github.com/cri-o/ocicni/pkg/ocicni/ocicni.go:253 +0x3d9

goroutine 39 [runnable]:
github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.(*Watcher).readEvents(0xc0000a3db0)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:172 fp=0xc00005a7d8 sp=0xc00005a7d0 pc=0x556e4eea3a30
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005a7e0 sp=0xc00005a7d8 pc=0x556e4df5fbe1
created by github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify.NewWatcher
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/github.com/fsnotify/fsnotify/inotify.go:59 +0x1aa

goroutine 25 [syscall]:
runtime.notetsleepg(0x556e512e48c0, 0xffffffffffffffff, 0xc00005afc8)
	/usr/lib/golang/src/runtime/lock_futex.go:227 +0x38 fp=0xc00005af98 sp=0xc00005af68 pc=0x556e4df0fad8
os/signal.signal_recv(0x0)
	/usr/lib/golang/src/runtime/sigqueue.go:139 +0x9e fp=0xc00005afc0 sp=0xc00005af98 pc=0x556e4df485ae
os/signal.loop()
	/usr/lib/golang/src/os/signal/signal_unix.go:23 +0x24 fp=0xc00005afe0 sp=0xc00005afc0 pc=0x556e4eb47814
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005afe8 sp=0xc00005afe0 pc=0x556e4df5fbe1
created by os/signal.init.0
	/usr/lib/golang/src/os/signal/signal_unix.go:29 +0x43

goroutine 24 [chan receive]:
runtime.gopark(0x556e4ff43d70, 0xc00001cef8, 0x7effadd1170d, 0x3)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00005b6d0 sp=0xc00005b6b0 pc=0x556e4df33cd5
runtime.goparkunlock(...)
	/usr/lib/golang/src/runtime/proc.go:307
runtime.chanrecv(0xc00001cea0, 0xc00005b7b0, 0xc0000a3d01, 0xc00001cea0)
	/usr/lib/golang/src/runtime/chan.go:524 +0x2ee fp=0xc00005b760 sp=0xc00005b6d0 pc=0x556e4df0abbe
runtime.chanrecv2(0xc00001cea0, 0xc00005b7b0, 0x556e4e61501b)
	/usr/lib/golang/src/runtime/chan.go:411 +0x2b fp=0xc00005b790 sp=0xc00005b760 pc=0x556e4df0a8bb
github.com/containers/libpod/vendor/k8s.io/klog.(*loggingT).flushDaemon(0x556e512b8840)
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:1010 +0x8d fp=0xc00005b7d8 sp=0xc00005b790 pc=0x556e4e7cb75d
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005b7e0 sp=0xc00005b7d8 pc=0x556e4df5fbe1
created by github.com/containers/libpod/vendor/k8s.io/klog.init.0
	/home/abuild/rpmbuild/BUILD/libpod-1.8.2/_build/src/github.com/containers/libpod/vendor/k8s.io/klog/klog.go:411 +0xd8

goroutine 21 [GC worker (idle)]:
runtime.gopark(0x556e4ff43c08, 0xc000476ce0, 0x1417, 0x0)
	/usr/lib/golang/src/runtime/proc.go:301 +0xf5 fp=0xc00005bf60 sp=0xc00005bf40 pc=0x556e4df33cd5
runtime.gcBgMarkWorker(0xc00004c000)
	/usr/lib/golang/src/runtime/mgc.go:1836 +0x105 fp=0xc00005bfd8 sp=0xc00005bf60 pc=0x556e4df20ff5
runtime.goexit()
	/usr/lib/golang/src/runtime/asm_amd64.s:1337 +0x1 fp=0xc00005bfe0 sp=0xc00005bfd8 pc=0x556e4df5fbe1
created by runtime.gcBgMarkStartWorkers
	/usr/lib/golang/src/runtime/mgc.go:1784 +0x79

Describe the results you expected:
podman should show status

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:            1.8.2
RemoteAPI Version:  1
Go Version:         go1.12.12
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: ""
  go version: go1.12.12
  podman version: 1.8.2
host:
  BuildahVersion: 1.14.3
  CgroupVersion: v1
  Conmon:
    package: conmon-2.0.14-1.1.el8.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.14, commit: eb75bd6e9fa7fb638b7b0eccc7fff9fa18358280'
  Distribution:
    distribution: '"centos"'
    version: "8"
  MemFree: 2494025728
  MemTotal: 3861200896
  OCIRuntime:
    name: runc
    package: runc-1.0.0-15.2.el8.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc10
      commit: db2349efc4dc0001462089382d175ed38fdba742
      spec: 1.0.1-dev
  SwapFree: 0
  SwapTotal: 0
  arch: amd64
  cpus: 1
  eventlogger: journald
  hostname: server01
  kernel: 5.5.13-1.el8.elrepo.x86_64
  os: linux
  rootless: false
  uptime: 20m 8.45s
registries:
  search:
  - registry.fedoraproject.org
  - registry.access.redhat.com
  - registry.centos.org
  - docker.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 4
  GraphDriverName: overlay
  GraphOptions:
    overlay.mountopt: nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 3
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

Package info (e.g. output of rpm -q podman or apt list podman):

podman-1.8.2-1.1.el8.x86_64

Additional environment details (AWS, VirtualBox, physical, etc.):
Google Cloud

CentOS Linux release 8.1.1911 (Core)
kernel-ml 5.5.13-1.el8.elrepo.x86_64 (same problem happening with centos 8 default kernel as well)
@openshift-ci-robot openshift-ci-robot added the kind/bug Categorizes issue or PR as related to a bug. label Mar 29, 2020
@github-actions
Copy link

A friendly reminder that this issue had no activity for 30 days.

@mheon
Copy link
Member

mheon commented Apr 30, 2020

Will try and take a look at this next week, once things are a bit more stable in master

@mheon
Copy link
Member

mheon commented Jun 2, 2020

Still repros on master, fixing now

@mheon mheon mentioned this issue Jun 2, 2020
Merged
mheon added a commit to mheon/libpod that referenced this issue Jun 2, 2020
@mheon
Properly follow linked namespace container for stats
42505f6 
Podman containers can specify that they get their network
namespace from another container. This is automatic in pods, but
any container can do it.

The problem is that these containers are not guaranteed to have a
network namespace of their own; it is perfectly valid to join the
network namespace of a --net=host container, and both containers
will end up in the host namespace. The code for obtaining network
stats did not account for this, and could cause segfaults as a
result. Fortunately, the fix is simple - the function we use to
get said stats already performs appropriate checks, so we just
need to recursively call it.

Fixes containers#5652

Signed-off-by: Matthew Heon <matthew.heon@pm.me>
@github-actions github-actions bot added the locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. label Sep 23, 2023
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Sep 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@mheon mheon

Labels
kind/bug Categorizes issue or PR as related to a bug. locked - please file new issue/PR Assist humans wanting to comment on an old issue or PR with locked comments. stale-issue
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Properly follow linked namespace container for stats mheon/libpod
3 participants
@nileshgr @mheon @openshift-ci-robot