Test failing for Duplicate lines in journal related to tempfiles.d

[jmarrero]

Describe the bug

Possible regression or change of behavior in the stack as the test for #26 is suddenly failing.

https://github.com/coreos/rpm-ostree/blob/main/tests/kolainst/nondestructive/misc.sh#L46

Reproduction steps

Run the kola test: https://github.com/coreos/rpm-ostree/blob/main/tests/kolainst/nondestructive/misc.sh

Expected behavior

Should not find duplicate /var/log

Aug 11 01:34:04 qemu0 kola-runext-misc.sh[2246]: Aug 11 01:33:29 localhost systemd-tmpfiles[332]: /usr/lib/tmpfiles.d/var.conf:14: Duplicate line for path "/var/log", ignoring.

Actual behavior

It find the Duplicate line.

System details

2024.7 ( but observed happening on CI in 2024.6 too)

Additional information

No response

[jmarrero]

@HuijingHei can you look at this one?

[HuijingHei]

The issue is introduced by dracut 101-1.fc40.x86_64 → 102-2.fc40.x86_64, and the error happened during initramfs, /var/log is defined both in dracut-tmpfiles.conf and var.conf

• The error is not on 40.20240717.20.1
• The error is on 40.20240722.20.0

When start FCOS vm, can not see the duplicated line /var/log under files in /usr/lib/tmpfiles.d/, then use command cosa run --qemu-image xx --kargs rd.break -c to start Dracut Emergency Shell, and find /var/log is defined both in dracut-tmpfiles.conf and var.conf, that causes the Duplicate line for path "/var/log"

Welcome to Fedora CoreOS 40.20240722.20.0 dracut-102-2.fc40 (Initramfs)!
...
sh-5.2# journalctl -u systemd-tmpfiles-setup.service |cat 
Aug 13 08:37:28 localhost systemd-tmpfiles[479]: /usr/lib/tmpfiles.d/var.conf:14: Duplicate line for path "/var/log", ignoring.
sh-5.2# cd /usr/lib/tmpfiles.d/
sh-5.2# grep -rn "/var/log" *
dracut-tmpfiles.conf:3:L /var/log           -    -    -    - ../run/initramfs/log
...
var.conf:14:d /var/log 0755 - - -

var.conf is included by systemd and it is not included in initramfs if start 40.20240717.20.1, but included in initramfs if start 40.20240722.20.0, there are something changed when building initramfs, and also see https://github.com/dracut-ng/dracut-ng/releases/tag/102, but got no hints.

[HuijingHei]

Confirm it is introduced by dracut 101-1.fc40.x86_64 → 102-2.fc40.x86_64.

Start vm with 40.20240717.20.1(no such error), then upgrade dracut to 102-2.fc40.x86_64, and regenerate initramfs using rpm-ostree initramfs --enable, reboot and get the "Duplicate line" error.

To fix the issue:

• exclude var.conf in initramfs, (it is not necessary and worked well without it in the past), need to create PR to dracut to remove line https://github.com/dracut-ng/dracut-ng/blob/main/modules.d/01systemd-tmpfiles/module-setup.sh#L41
• ignore the error as it is harmless, need to update the CI code, maybe ignore like journalctl --grep 'Duplicate line' | grep -v 'Duplicate line for path "/var/log"' in https://github.com/coreos/rpm-ostree/blob/main/tests/kolainst/nondestructive/misc.sh#L45

I think we can update the CI code to ignore it first, and check if we should remove var.conf in initramfs, any suggestions? @jmarrero @jlebon

[jlebon]

If the warning is coming from files not owned by us, then we should just ignore it. In this case, it sounds like it boils down to just ignoring log entries from before switchroot when doing this check.

[HuijingHei]

Thanks @jlebon for the pointer, create PR #5049 to fix the issue.