Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[rawhide] kernel 6.9 regressions causing podman test failures #1693

Closed
dustymabe opened this issue Mar 15, 2024 · 2 comments
Closed

[rawhide] kernel 6.9 regressions causing podman test failures #1693

dustymabe opened this issue Mar 15, 2024 · 2 comments

Comments

@dustymabe
Copy link
Member

Right now there are two test failures that appear to be related to the new 6.9 kernel RC being introduced in rawhide:

  • ext.config.podman.rootless-systemd
  • ext.config.podman.rootless-pasta-networking
  • ext.config.podman.dns
[2024-03-15T19:54:30.071Z]     --- FAIL: non-exclusive-test-bucket-0/ext.config.podman.rootless-pasta-networking (181.24s)
[2024-03-15T19:54:30.071Z]             harness.go:106: TIMEOUT[3m0s]: ssh: sudo ./kolet run-test-unit --deny-reboots kola-runext-59.service
[2024-03-15T19:54:30.071Z]             harness.go:106: TIMEOUT[3m0s]: ssh: journalctl -t kola-runext-rootless-pasta-networking
[2024-03-15T19:54:30.071Z]     --- FAIL: non-exclusive-test-bucket-0/ext.config.podman.dns (181.64s)
[2024-03-15T19:54:30.071Z]             harness.go:106: TIMEOUT[3m0s]: ssh: sudo ./kolet run-test-unit --deny-reboots kola-runext-58.service
[2024-03-15T19:54:30.071Z]             harness.go:106: TIMEOUT[3m0s]: ssh: journalctl -t kola-runext-test.sh
[2024-03-15T20:03:36.529Z] --- FAIL: ext.config.podman.rootless-systemd (903.93s)
[2024-03-15T20:03:36.529Z]         harness.go:106: TIMEOUT[15m0s]: ssh: sudo ./kolet run-test-unit kola-runext.service
[2024-03-15T20:03:36.529Z]         harness.go:106: TIMEOUT[15m0s]: ssh: journalctl -t kola-runext-rootless-systemd

The pasta networking one seems to have been already investigated over in containers/podman#22052

The dns failure is easily reproducible. It just hangs:

[core@cosa-devsh ~]$ podman network create testnetwork
testnetwork
[core@cosa-devsh ~]$ podman run --rm -t --network=testnetwork quay.io/fedora/fedora:39 getent hosts google.com
Trying to pull quay.io/fedora/fedora:39...
Getting image source signatures
Copying blob 718a00fe3212 done   |
Copying config 368a084ba1 done   |
Writing manifest to image destination

I'm not sure about the rootless systemd failure for now.

The logs show it just building and never finishing:

Mar 15 19:18:39.687084 session-c1.scope[2143]: + podman build -t httpd .
Mar 15 19:18:39.836453 session-c1.scope[2152]: shadow: unknown configuration item 'PASS_MIN_LEN' in '/etc/login.defs'
Mar 15 19:18:39.837355 session-c1.scope[2152]: shadow: unknown configuration item 'PASS_CHANGE_TRIES' in '/etc/login.defs'
Mar 15 19:18:39.837362 session-c1.scope[2152]: shadow: unknown configuration item 'PASS_ALWAYS_WARN' in '/etc/login.defs'
Mar 15 19:18:39.837367 session-c1.scope[2152]: shadow: unknown configuration item 'HMAC_CRYPTO_ALGO' in '/etc/login.defs'
Mar 15 19:18:39.842412 session-c1.scope[2155]: shadow: unknown configuration item 'PASS_MIN_LEN' in '/etc/login.defs'
Mar 15 19:18:39.842699 session-c1.scope[2155]: shadow: unknown configuration item 'PASS_CHANGE_TRIES' in '/etc/login.defs'
Mar 15 19:18:39.842706 session-c1.scope[2155]: shadow: unknown configuration item 'PASS_ALWAYS_WARN' in '/etc/login.defs'
Mar 15 19:18:39.842711 session-c1.scope[2155]: shadow: unknown configuration item 'HMAC_CRYPTO_ALGO' in '/etc/login.defs'
Mar 15 19:18:39.998840 kernel: evm: overlay not supported
Mar 15 19:18:40.009689 session-c1.scope[2151]: 2024-03-15 19:18:40.009421523 +0000 UTC m=+0.128251481 system refresh
Mar 15 19:18:40.017435 user@1000.service[1737]: Created slice session.slice - User Core Session Slice.
Mar 15 19:18:40.027121 user@1000.service[1737]: Starting dbus-broker.service - D-Bus User Message Bus...
Mar 15 19:18:40.076412 user@1000.service[2169]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +31: Eavesdropping is deprecated and ignored
Mar 15 19:18:40.077513 user@1000.service[2169]: Policy to allow eavesdropping in /usr/share/dbus-1/session.conf +33: Eavesdropping is deprecated and ignored
Mar 15 19:18:40.078142 user@1000.service[1737]: Started dbus-broker.service - D-Bus User Message Bus.
Mar 15 19:18:40.083421 user@1000.service[2169]: Ready
Mar 15 19:18:40.092286 user@1000.service[1737]: Created slice user.slice - Slice /user.
Mar 15 19:18:40.109365 user@1000.service[1737]: Started podman-2151.scope.
Mar 15 19:18:40.116107 kola-runext.service[2141]: STEP 1/3: FROM quay.io/fedora/fedora:39
Mar 15 19:18:40.121689 user@1000.service[2151]: Trying to pull quay.io/fedora/fedora:39...
Mar 15 19:18:40.557043 user@1000.service[2151]: Getting image source signatures
Mar 15 19:18:40.558329 user@1000.service[2151]: Copying blob sha256:718a00fe32127ad01ddab9fc4b7c968ab2679c92c6385ac6865ae6e2523275e4
Mar 15 19:18:46.183137 user@1000.service[2151]: Copying config sha256:368a084ba17dcba88f5b23acfa47481131010219524fd9c41af87d709a04845b
Mar 15 19:18:46.186799 user@1000.service[2151]: Writing manifest to image destination
Mar 15 19:18:46.211849 kola-runext.service[2141]: STEP 2/3: RUN dnf -y update && dnf -y install systemd httpd && dnf clean all && systemctl enable httpd
Mar 15 19:18:46.410970 user@1000.service[1737]: Started crun-buildah-buildah3509006848.scope - libcrun container.
Mar 15 19:18:46.455813 kernel: tun: Universal TUN/TAP device driver, 1.6
Mar 15 19:18:56.341620 init.scope[1]: systemd-hostnamed.service: Deactivated successfully.
Mar 15 19:18:56.342000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=systemd-hostnamed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 15 19:18:56.351000 audit: BPF prog-id=68 op=UNLOAD
Mar 15 19:18:56.351000 audit: BPF prog-id=67 op=UNLOAD
Mar 15 19:18:56.351000 audit: BPF prog-id=66 op=UNLOAD
Mar 15 19:19:30.242643 init.scope[1]: rpm-ostreed.service: Deactivated successfully.
Mar 15 19:19:30.242000 audit[1]: SERVICE_STOP pid=1 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:init_t:s0 msg='unit=rpm-ostreed comm="systemd" exe="/usr/lib/systemd/systemd" hostname=? addr=? terminal=? res=success'
Mar 15 19:23:50.116342 user@1000.service[1737]: Created slice background.slice - User Background Tasks Slice.
Mar 15 19:23:50.122859 user@1000.service[1737]: Starting systemd-tmpfiles-clean.service - Cleanup of User's Temporary Files and Directories...
Mar 15 19:23:50.137957 user@1000.service[1737]: Finished systemd-tmpfiles-clean.service - Cleanup of User's Temporary Files and Directories.
dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Mar 16, 2024
@dustymabe
overrides: pin to kernel-6.8.0-63.fc41
eec5b11 
The 6.9 kernel is causing several podman tests to fail. We'll
investigate in coreos/fedora-coreos-tracker#1693
@dustymabe dustymabe mentioned this issue Mar 16, 2024
Merged
dustymabe added a commit to coreos/fedora-coreos-config that referenced this issue Mar 16, 2024
@dustymabe
overrides: pin to kernel-6.8.0-63.fc41
5efe808 
The 6.9 kernel is causing several podman tests to fail. We'll
investigate in coreos/fedora-coreos-tracker#1693
@dustymabe
Copy link
Member Author

pin in coreos/fedora-coreos-config@5efe808

@dustymabe dustymabe mentioned this issue Mar 18, 2024
Closed
@dustymabe
Copy link
Member Author

With the fix from containers/podman#22052 (comment) (i.e. passt-0^20240318.g615d370-1.fc41) it looks like the tests all pass again.

dustymabe added a commit to dustymabe/fedora-coreos-config that referenced this issue Mar 18, 2024
@dustymabe
overrides: fast-track passt; unpin kernel
2169351 
The new passt release allows us to unpin the kernel and still have
all podman tests pass. See
coreos/fedora-coreos-tracker#1693 (comment)
@dustymabe dustymabe mentioned this issue Mar 18, 2024
Merged
dustymabe added a commit to coreos/fedora-coreos-config that referenced this issue Mar 18, 2024
@dustymabe
overrides: fast-track passt; unpin kernel
1176524 
The new passt release allows us to unpin the kernel and still have
all podman tests pass. See
coreos/fedora-coreos-tracker#1693 (comment)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@dustymabe