NET IO BLOCK IO is Not Present via Running podman stats

[SAURABH110894]

/kind bug

Description

Steps to reproduce the issue:

1. podman stats

Describe the results you received:
ID NAME CPU % MEM USAGE / LIMIT MEM % NET IO BLOCK IO PIDS
99129948242c testpodman 0.11% 119.7MB / 16.8GB 0.71% -- / -- -- / -- 24

NET IO and Block IO Values is not coming

Describe the results you expected:

Additional information you deem important (e.g. issue happens only occasionally):

Output of podman version:

Version:      3.2.3
API Version:  3.2.3
Go Version:   go1.15.14
Built:        Wed Aug 11 02:25:16 2021
OS/Arch:      linux/amd64

Output of podman info --debug:

host:
  arch: amd64
  buildahVersion: 1.21.3
  cgroupControllers: []
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.0.29-1.module_el8.4.0+886+c9a8d9ad.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.0.29, commit: 97bba1e91aaab5be2e93bacd34ec4e66655a02ae'
  cpus: 2
  distribution:
    distribution: '"centos"'
    version: "8"
  eventLogger: file
  hostname: poc-devops.localhost.com
  idMappings:
    gidmap:
    - container_id: 0
      host_id: 1006
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
    uidmap:
    - container_id: 0
      host_id: 1006
      size: 1
    - container_id: 1
      host_id: 493216
      size: 65536
  kernel: 4.18.0-305.12.1.el8_4.x86_64
  linkmode: dynamic
  memFree: 14127460352
  memTotal: 16803004416
  ociRuntime:
    name: runc
    package: runc-1.0.0-74.rc95.module_el8.4.0+886+c9a8d9ad.x86_64
    path: /usr/bin/runc
    version: |-
      runc version spec: 1.0.2-dev
      go: go1.15.14
      libseccomp: 2.5.1
  os: linux
  remoteSocket:
    path: /run/user/1006/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_NET_RAW,CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: true
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: true
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.1.8-1.module_el8.4.0+641+6116a774.x86_64
    version: |-
      slirp4netns version 1.1.8
      commit: d361001f495417b880f20329121e3aa431a8f90f
      libslirp: 4.3.1
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.1
  swapFree: 17179865088
  swapTotal: 17179865088
  uptime: 14m 51.94s
registries:
  nexus.localhost.com:8084:
    Blocked: false
    Insecure: true
    Location: nexus.localhost.com:8084
    MirrorByDigestOnly: false
    Mirrors: []
    Prefix: nexus.localhost.com:8084
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /home/joe/.config/containers/storage.conf
  containerStore:
    number: 11
    paused: 0
    running: 1
    stopped: 10
  graphDriverName: overlay
  graphOptions:
    overlay.mount_program:
      Executable: /usr/bin/fuse-overlayfs
      Package: fuse-overlayfs-1.6-1.module_el8.4.0+886+c9a8d9ad.x86_64
      Version: |-
        fusermount3 version: 3.2.1
        fuse-overlayfs: version 1.6
        FUSE library version 3.2.1
        using FUSE kernel interface version 7.26
  graphRoot: /home/joe/.local/share/containers/storage
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "false"
  imageStore:
    number: 5
  runRoot: /run/user/1006/containers
  volumePath: /home/joe/.local/share/containers/storage/volumes
version:
  APIVersion: 3.2.3
  Built: 1628628916
  BuiltTime: Wed Aug 11 02:25:16 2021
  GitCommit: ""
  GoVersion: go1.15.14
  OsArch: linux/amd64
  Version: 3.2.3

Package info (e.g. output of rpm -q podman or apt list podman):

podman-3.2.3-0.10.module_el8.4.0+886+c9a8d9ad.x86_64

Have you tested with the latest version of Podman and have you checked the Podman Troubleshooting Guide? (https://github.com/containers/podman/blob/master/troubleshooting.md)

Yes/No

Yes

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical

[rhatdan]

@giuseppe PTAL

[mheon]

cgroupManager: systemd
cgroupVersion: v2
rootless: true

Are these controllers not enabled for rootless v2?

[SAURABH110894]

@mheon can you please help me how to verify whether above controller is rootless ?

[Luap99]

The code currently disables the networks stats for rootless users.

podman/libpod/networking_linux.go

Lines 912 to 938 in 420ff1d

	func getContainerNetIO(ctr *Container) (*netlink.LinkStatistics, error) {
	var netStats *netlink.LinkStatistics
	// With slirp4netns, we can't collect statistics at present.
	// For now, we allow stats to at least run by returning nil
	if rootless.IsRootless() || ctr.config.NetMode.IsSlirp4netns() {
	return netStats, nil
	}
	netNSPath, netPathErr := getContainerNetNS(ctr)
	if netPathErr != nil {
	return nil, netPathErr
	}
	if netNSPath == "" {
	// If netNSPath is empty, it was set as none, and no netNS was set up
	// this is a valid state and thus return no error, nor any statistics
	return nil, nil
	}
	err := ns.WithNetNSPath(netNSPath, func(_ ns.NetNS) error {
	// FIXME get the interface from the container netstatus
	link, err := netlink.LinkByName("eth0")
	if err != nil {
	return err
	}
	netStats = link.Attrs().Statistics
	return nil
	})
	return netStats, err
	}

I don't think this can work 100% correctly for rootless, the port forwarder uses a bit of a hack to get the data into the container. It will not send the data through the actual eth0 or tap0 interface.
The question is which net I/O should be shown, the data which is forwarded into the container or the data when the container connects to the outside?

Also when I look at this, it will not work for multiple interfaces at the moment. I think if we add the statistics of all interfaces together we could make it work.

[mheon]

Rootless networking is probably a lost cause. Storage might work, if the appropriate cgroups are available, I think.

[Luap99]

I tried it rootless works fine, the problem is that in order to get the port forwarding traffic we have to read the loopback adapter statistics. The loopback adapter on the other hand will not show the outgoing traffic. Those can be get via the tap0 adapter. I think in order to make it work we should show the sum of all adapters.
This would also make sense for containers with more than one network, at the moment this is hardcoded to only read eth0.

[SAURABH110894]

@Luap99 , Could you please help me how I can achieve this tasks ?
We have plan to migrate from docker to podman in prod server, Could you please help us for NET IO and Block IO Tasks to display the result via podman stats (with rootless user )?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 any progress on this?

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@Luap99 friendly ping.

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[rhatdan]

@giuseppe PTAL

[github-actions]

A friendly reminder that this issue had no activity for 30 days.

[giuseppe]

opened a PR to add network stats from slirp4netns: #13101

block IO stats already work for rootless on cgroup v2 when the io controller is delegated to the user.