[Bug]: `systemctl status` failed after upgrading systemd to version 250 #31

winderica · 2022-01-06T17:28:10Z

Describe the bug

After upgrading the version of systemd to 250, systemctl status doesn't work in Distrod and shows Failed to dump process list for 'PC', ignoring: Input/output error.

Steps to reproduce

Upgrade systemd to version 250.

❯  systemctl --version
systemd 250 (250.1-1-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

Run systemctl status, and the error is as below.

❯ sudo systemctl status
Failed to dump process list for 'PC', ignoring: Input/output error
● PC
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Fri 2022-01-07 00:21:11 CST; 41min ago
   CGroup: /

Downgrade systemd to 249 and it works again.

❯  systemctl --version
systemd 249 (249rc3-2-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified
❯ sudo systemctl status
● PC
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Fri 2022-01-07 00:21:11 CST; 41min ago
   CGroup: /
           ├─init.scope
           └─system.slice
             ├─systemd-networkd.service
             │ └─922 /usr/lib/systemd/systemd-networkd
             ├─systemd-udevd.service
             ├─systemd-homed.service
             │ └─129 /usr/lib/systemd/systemd-homed
             ├─cronie.service
             │ └─116 /usr/bin/crond -n
             ├─docker.service …
             │ ├─125 /usr/bin/dockerd -H fd://
             │ └─209 containerd --config /var/run/docker/containerd/containerd.toml --log-level info
             ├─systemd-journald.service
             ├─sshd.service
             │ └─128 sshd: /usr/bin/sshd -D [listener] 0 of 10-100 startups
             ├─systemd-userdbd.service
             │ ├─1062 /usr/lib/systemd/systemd-userdbd
             │ ├─1178 systemd-userwork
             │ ├─1179 systemd-userwork
             │ └─1180 systemd-userwork

Expected behavior

systemctl status shouldn't fail. Actually, version 250 works fine on my another Arch Linux machine.

Windows version

Windows 11 (Build 22523.1000)

Linux kernel version

Linux PC 5.10.74.3-microsoft-standard-WSL2 #1 SMP Mon Oct 18 19:27:44 UTC 2021 x86_64 GNU/Linux

Distro

Arch Linux

How did you install that distro?

Installed by Distrod wizard

Logs

[Distrod][DEBUG] distrod-exec: exec_command_in_distro
[Distrod][DEBUG] starting /init from distrod-exec
[Distrod][DEBUG] WSL envs: "WSLENV" = "WT_SESSION::WT_PROFILE_ID"
[Distrod][DEBUG] WSL envs: "WSL_DISTRO_NAME" = "Distrod"
[Distrod][DEBUG] WSL envs: "WSL_INTEROP" = "/run/WSL/8_interop"
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/run/distrod/cmdline")), target: ContainerPath("/proc/cmdline"), fstype: None, flags: MS_BIND, is_file: true 
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd"
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd/system"
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/systemd/system/portproxy.service"
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/opt/distrod/run/systemd/system/portproxy.service")), target: ContainerPath("/run/systemd/system/portproxy.service"), fstype: None, flags: MS_BIND, is_file: true
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/tmpfiles.d"
[Distrod][TRACE] mount_distrod_run_files: path: "/opt/distrod/run/tmpfiles.d/x11.conf"
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/opt/distrod/run/tmpfiles.d/x11.conf")), target: ContainerPath("/run/tmpfiles.d/x11.conf"), fstype: None, flags: MS_BIND, is_file: true
[Distrod][DEBUG] DistroLauncher::launch
[Distrod][DEBUG] Container::with_mount source: Some(HostPath("/run/distrod/distrod_wsl_env-uid1000")), target: ContainerPath("/run/distrod/distrod_wsl_env-uid1000"), fstype: None, flags: MS_BIND, is_file: true
[Distrod][DEBUG] Spawning the command or the waiter.
[Distrod][DEBUG] Executing a command in the distro.
[Distrod][DEBUG] Failed to ignore signal Sys(EINVAL)
[Distrod][DEBUG] Failed to ignore signal Sys(EINVAL)
[Distrod][DEBUG] Distro::exec_command.
[Distrod][DEBUG] Container::exec_command.
[Distrod][TRACE] mounting source: Some(
    ContainerPath(
        "/run/distrod/cmdline",
    ),
), mount: ContainerMount { source: Some(HostPath("/run/distrod/cmdline")), target: ContainerPath("/proc/cmdline"), fstype: None, flags: MS_BIND, data: None, is_file: true }
[Distrod][DEBUG] dropping privilege. kmsg logging in the child ends here.
[Distrod][DEBUG] Triple fork done.
[Distrod][TRACE] mounting source: Some(
    ContainerPath(
        "/opt/distrod/run/systemd/system/portproxy.service",
    ),
), mount: ContainerMount { source: Some(HostPath("/opt/distrod/run/systemd/system/portproxy.service")), target: ContainerPath("/run/systemd/system/portproxy.service"), fstype: None, flags: MS_BIND, data: None, is_file: true }
[Distrod][TRACE] mounting source: Some(
    ContainerPath(
        "/opt/distrod/run/tmpfiles.d/x11.conf",
    ),
), mount: ContainerMount { source: Some(HostPath("/opt/distrod/run/tmpfiles.d/x11.conf")), target: ContainerPath("/run/tmpfiles.d/x11.conf"), fstype: None, flags: MS_BIND, data: None, is_file: true }
[Distrod][DEBUG] The parent of the second of three forks exits.
[Distrod][TRACE] skipping an identical mount: Some(
    ContainerPath(
        "/run/distrod/distrod_wsl_env-uid1000",
    ),
), ContainerMount {
    source: Some(
        HostPath(
            "/run/distrod/distrod_wsl_env-uid1000",
        ),
    ),
    target: ContainerPath(
        "/run/distrod/distrod_wsl_env-uid1000",
    ),
    fstype: None,
    flags: MS_BIND,
    data: None,
    is_file: true,
}
[Distrod][DEBUG] Spawning the command or the waiter.
[Distrod][DEBUG] Spawning the waiter.
[Distrod][DEBUG] Failed to ignore signal Sys(EINVAL)
[Distrod][DEBUG] Failed to ignore signal Sys(EINVAL)

additional comment

No response

The text was updated successfully, but these errors were encountered:

nullpo-head · 2022-01-15T14:55:33Z

Thanks for reporting! I'll look into it when I have time. (I'm busy right now due to house moving...

DylanSimowitz · 2022-01-17T07:37:39Z

I'm experiencing the same issue. To add, systemctl --user status outputs

Failed to connect to bus: No such file or directory

jueti · 2022-02-05T04:09:07Z

Same issue.

Test fine when downgrading systemd version to systemd 249 (249.7-2-arch).

I got a conflict with hwids and hwdata.
I just easily delete conflicting files:

$ cp /usr/share/hwdata/pci.ids .
$ cp /usr/share/hwdata/pnp.ids .
$ cp /usr/share/hwdata/usb.ids .
$ sudo rm /usr/share/hwdata/pci.ids /usr/share/hwdata/pnp.ids /usr/share/hwdata/usb.ids

Now, I am success for downgrading:

$ sudo downgrade hwids
loading packages...
resolving dependencies...
looking for conflicting packages...

Packages (1) hwids-20201207-1

Total Installed Size:  1.91 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                                           [########################################] 100%
(1/1) checking package integrity                                         [########################################] 100%
(1/1) loading package files                                              [########################################] 100%
(1/1) checking for file conflicts                                        [########################################] 100%
(1/1) checking available disk space                                      [########################################] 100%
:: Processing package changes...
(1/1) installing hwids                                                   [########################################] 100%
:: Running post-transaction hooks...
(1/1) Arming ConditionNeedsUpdate...
add hwids to IgnorePkg? [y/N] y

$ sudo downgrade systemd
loading packages...
warning: downgrading package systemd (250.3-2 => 249.7-2)
resolving dependencies...
looking for conflicting packages...

Packages (1) systemd-249.7-2


Total Installed Size:  25.80 MiB
Net Upgrade Size:      -2.60 MiB

:: Proceed with installation? [Y/n] y
(1/1) checking keys in keyring                                           [########################################] 100%
(1/1) checking package integrity                                         [########################################] 100%
(1/1) loading package files                                              [########################################] 100%
(1/1) checking for file conflicts                                        [########################################] 100%
(1/1) checking available disk space                                      [########################################] 100%
warning: could not get file information for usr/lib/systemd/system/system-systemd\x2dcryptsetup.slice
:: Processing package changes...
(1/1) downgrading systemd                                                [########################################] 100%
:: Running post-transaction hooks...
(1/9) Creating system user accounts...
(2/9) Updating journal message catalog...
(3/9) Reloading system manager configuration...
(4/9) Updating udev hardware database...
(5/9) Applying kernel sysctl settings...
Couldn't write '16' to 'kernel/sysrq', ignoring: No such file or directory
(6/9) Creating temporary files...
(7/9) Reloading device manager configuration...
(8/9) Arming ConditionNeedsUpdate...
(9/9) Reloading system bus configuration...

systemd 249 (249.7-2-arch) works well.

$ systemctl --version
systemd 249 (249.7-2-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

$ systemctl status
● UD3
    State: running
     Jobs: 1 queued
   Failed: 0 units
    Since: Sat 2022-02-05 11:49:42 CST; 15min ago
   CGroup: /
           ├─init.scope
           │ └─1 /usr/lib/systemd/systemd systemd.setenv=WSL_INTEROP=/run/WSL/8_interop systemd.setenv=WSLENV=WT_SESSIO>
           └─system.slice
             ├─systemd-udevd.service
             │ └─41 /usr/lib/systemd/systemd-udevd
             ├─systemd-journald.service
             │ └─31 /usr/lib/systemd/systemd-journald
             ├─dbus.service
             │ └─59 /usr/bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation --syslog>
             ├─systemd-tmpfiles-clean.service
             │ └─2569 systemd-tmpfiles --clean
             └─systemd-logind.service
               └─60 /usr/lib/systemd/systemd-logind
[jason@UD3 18200]$

May be:

systemd 250-1.2 -> 250-1.1 - "hwids" was replaced by "hwdata"

cerebrate · 2022-02-14T18:11:06Z

This may be this:

systemd/systemd#22089

axieum · 2022-03-26T04:51:47Z

I am also encountering this on systemd/systemd@v250 -

$ systemctl --version; systemctl status
systemd 250 (250.4-2-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified
Failed to dump process list for 'Adair', ignoring: Input/output error
● Adair
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Sat 2022-03-26 14:20:25 AEST; 29min ago
   CGroup: /

nullpo-head · 2022-03-26T16:21:37Z

Hi, thanks for the reports, all. I was busy recently but now started to work on distrod fixes again. Actually, I already figured out the root cause and am working on fix. The bug happened only on weird Linux set up (yes, it's WSL), so I'm doing fix from distrod side; making a cgroup namespace. Please wait for a while!

winderica · 2022-05-23T08:47:44Z

This may be this:

systemd/systemd#22089

It still does not work after upgrading systemd to version 251. FYI:

❯ systemctl --version
systemd 251 (251-1-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +TPM2 +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified

❯ sudo systemctl status
Failed to dump process list for 'PC', ignoring: Input/output error
● PC
    State: running
    Units: 245 loaded (incl. loaded aliases)
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2022-05-23 11:39:33 CST; 5h 6min ago
  systemd: 251-1-arch
  Tainted: cgroupsv1
   CGroup: /

SuperSandro2000 · 2022-06-08T23:11:50Z

This may be this:

systemd/systemd#22089

Tried that patch on top of 250.4 on NixOS and also didn't work.

SuperSandro2000 · 2022-06-09T12:28:10Z

strace

ioctl(1, TIOCGWINSZ, {ws_row=43, ws_col=190, ws_xpixel=0, ws_ypixel=0}) = 0
sendmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\1\4\1\f\0\0\0\3\0\0\0\247\0\0\0\1\1o\0\31\0\0\0/org/freedesktop/systemd1\0\0\0\0\0\0\0\3\1s\0\20\0\0\0GetUnitProcesses\0\0\0\0\0\0\0\0\2\1s\0 \0\0\0org.freedesktop.systemd1.Manager\0\0\0\0\0\0\0\0\6\1s\0\30\0\0\0org.freedesktop.systemd1\0\0\0\0\0\0\0\0\10\1g\0\1s\0\0", iov_len=184}, {iov_base="\7\0\0\0-.slice\0", iov_len=12}], msg_iovlen=2, msg_controllen=0, msg_flags=0}, MSG_DONTWAIT|MSG_NOSIGNAL) = 196
clock_gettime(CLOCK_MONOTONIC, {tv_sec=207, tv_nsec=180553963}) = 0
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="l\3\1\1\27\0\0\0O\1\0\0]\0\0\0\5\1u\0\3\0\0\0", iov_len=24}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 24
recvmsg(3, {msg_name=NULL, msg_namelen=0, msg_iov=[{iov_base="\6\1s\0\5\0\0\0:1.17\0\0\0\4\1s\0\"\0\0\0org.freedesktop.DBus.Error.IOError\0\0\0\0\0\0\10\1g\0\1s\0\0\7\1s\0\4\0\0\0:1.1\0\0\0\0\22\0\0\0Input/output error\0", iov_len=111}], msg_iovlen=1, msg_controllen=0, msg_flags=MSG_CMSG_CLOEXEC}, MSG_DONTWAIT|MSG_CMSG_CLOEXEC) = 111
writev(2, [{iov_base="Failed to dump process list for 'W-PF2ZA0JC', ignoring: Input/output error", iov_len=74}, {iov_base="\n", iov_len=1}], 2Failed to dump process list for 'W-PF2ZA0JC', ignoring: Input/output error

mangkoran · 2022-09-12T02:41:28Z

I made a bit of progress debugging this issue. I followed this post, executing sudo loginctl enable-linger $USER.

Then, I read this post, and noticed that DBUS_SESSION_BUS_ADDRESS is not set too. Set it using export DBUS_SESSION_BUS_ADDRESS=/run/user/$(id -u $USER)/bus

After that, systemctl should run "normally".

Edit: Removed OOT part. You can always see the history 😉

cerebrate · 2022-09-23T18:16:47Z

It may be worthy of note that this same failure happens when using Microsoft's new built-in systemd support.

v1nh1shungry mentioned this issue Jul 24, 2022

Broken systemctl status arkane-systems/genie#292

Closed

1 task

cerebrate mentioned this issue Sep 25, 2022

systemd status fails output on Ubuntu microsoft/WSL#8879

Open

2 tasks

matt335672 mentioned this issue Jun 13, 2023

xrdp.service: Can't open PID file /run/xrdp/xrdp.pid (yet?) after start: Operation not permitted neutrinolabs/xrdp#2728

Closed

trallnag mentioned this issue Apr 29, 2024

cgroup-util: make sure cg_read_pid() can deal sanely with unmapped PIDs (as in foreign pidns) systemd/systemd#32534

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Bug]: `systemctl status` failed after upgrading systemd to version 250 #31

[Bug]: `systemctl status` failed after upgrading systemd to version 250 #31

winderica commented Jan 6, 2022

nullpo-head commented Jan 15, 2022

DylanSimowitz commented Jan 17, 2022

jueti commented Feb 5, 2022 •

edited

Loading

cerebrate commented Feb 14, 2022

axieum commented Mar 26, 2022

nullpo-head commented Mar 26, 2022

winderica commented May 23, 2022 •

edited

Loading

SuperSandro2000 commented Jun 8, 2022

SuperSandro2000 commented Jun 9, 2022

mangkoran commented Sep 12, 2022 •

edited

Loading

cerebrate commented Sep 23, 2022

[Bug]: systemctl status failed after upgrading systemd to version 250 #31

[Bug]: systemctl status failed after upgrading systemd to version 250 #31

Comments

winderica commented Jan 6, 2022

Describe the bug

Steps to reproduce

Expected behavior

Windows version

Linux kernel version

Distro

How did you install that distro?

Logs

additional comment

nullpo-head commented Jan 15, 2022

DylanSimowitz commented Jan 17, 2022

jueti commented Feb 5, 2022 • edited Loading

cerebrate commented Feb 14, 2022

axieum commented Mar 26, 2022

nullpo-head commented Mar 26, 2022

winderica commented May 23, 2022 • edited Loading

SuperSandro2000 commented Jun 8, 2022

SuperSandro2000 commented Jun 9, 2022

mangkoran commented Sep 12, 2022 • edited Loading

cerebrate commented Sep 23, 2022

[Bug]: `systemctl status` failed after upgrading systemd to version 250 #31

[Bug]: `systemctl status` failed after upgrading systemd to version 250 #31

jueti commented Feb 5, 2022 •

edited

Loading

winderica commented May 23, 2022 •

edited

Loading

mangkoran commented Sep 12, 2022 •

edited

Loading