CI: podman logs k8s-file never seeing output

[edsantiago]

The common factor seems to be:

$ podman logs ...

EXPECTED xxxx + yyyy
ACTUAL: less than that

This is probably a bug in the tests, not a podman bug. And I don't think it's the same as the other log timing bug tracked in #23479, but it might be.

• debian-13 : sys podman debian-13 rootless host sqlite
  • 08-13 16:53 in [sys] |035| podman logs - --since --follow k8s-file
• fedora-39 : sys podman fedora-39 root host boltdb
  • 08-13 22:48 in [sys] |035| podman logs - --since --follow k8s-file
  • 08-12 18:38 in [sys] |035| podman logs - --since --follow k8s-file
  • 08-12 15:10 in [sys] |035| podman logs - --since --follow k8s-file
• rawhide : sys podman rawhide root host sqlite
  • 08-13 21:26 in [sys] |035| podman logs - multi k8s-file

x	x	x	x	x	x
sys(5)	podman(5)	fedora-39(3)	root(4)	host(5)	boltdb(3)
		rawhide(1)	rootless(1)		sqlite(2)
		debian-13(1)

[Luap99]

The file logger is known to miss lines #21914 (when using --follow/-f)

I wonder if I should just fork the tail lib and write our own purpose build thing to fix these given upstream is not active at all.

[edsantiago]

Just triggered in a non-parallel test. I've edited the description.

[edsantiago]

Current list of hits. Almost all are in my parallel PR, but (as of now) not all.

• debian-13 : sys podman debian-13 root host sqlite
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-15 10:31 in [sys] |035| podman logs - --since --follow k8s-file
• debian-13 : sys podman debian-13 rootless host sqlite
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-13 16:53 in [sys] |035| podman logs - --since --follow k8s-file
• fedora-39 : sys podman fedora-39 root host boltdb
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-13 22:48 in [sys] |035| podman logs - --since --follow k8s-file
    • 08-12 18:38 in [sys] |035| podman logs - --since --follow k8s-file
    • 08-12 15:10 in [sys] |035| podman logs - --since --follow k8s-file
• fedora-39 : sys podman fedora-39 rootless host boltdb
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-14 07:47 in [sys] |035| podman logs - --since --follow k8s-file
• fedora-39 : sys remote fedora-39 root host boltdb [remote]
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-15 18:08 in [sys] |035| podman logs - --since --follow k8s-file
• fedora-40 : sys podman fedora-40 root host sqlite
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-16 17:52 in [sys] |035| podman logs - --since --follow k8s-file
• fedora-40 : sys podman fedora-40 rootless host sqlite
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-16 17:53 in [sys] |035| podman logs - --since --follow journald
• fedora-40 : sys remote fedora-40 rootless host sqlite [remote]
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-15 20:27 in [sys] |035| podman logs - --since --follow k8s-file
• fedora-40-aarch64 : sys remote fedora-40-aarch64 root host sqlite [remote]
  • PR update github.com/vishvananda/netlink to v1.3.0 #23749
    • 08-26 08:04 in [sys] [035] podman logs - --since --follow k8s-file
• rawhide : sys podman rawhide root host sqlite
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-13 21:26 in [sys] |035| podman logs - multi k8s-file
• rawhide : sys podman rawhide rootless host sqlite
  • PR WIP: system test parallelization: two-pass approach #23275
    • 08-16 19:39 in [sys] |035| podman logs - --since --follow journald

x	x	x	x	x	x
sys(13)	podman(10)	fedora-39(5)	root(8)	host(13)	sqlite(8)
	remote(3)	fedora-40(3)	rootless(5)		boltdb(5)
		debian-13(2)
		rawhide(2)
		fedora-40-aarch64(1)

[Luap99]

First it seems it is only one failure in the last week after you applied logging fixes?!

But also I do not see the log file from ee851a1 in the logs in the failure from the 08-26?

[Luap99]

But also I do not see the log file from ee851a1 in the logs in the failure from the 08-26?

Oh wait this was not your PR. IMO my logging fork fix work given you didn't see this the last week per your logs above then

[edsantiago]

IMO my logging fork fix works

Is this df865c8 , your fix for nxadm/tail? ISTR seeing failures even after I cherrypicked it; and the timestamps confirm it. Or did you mean a different fix? I'm also carrying Jan's patches, and a few other changes, so it's really hard to say why the flake hasn't recurred since August 16.

[Luap99]

Is this df865c8 , your fix for nxadm/tail?

I think so, but if you have a ton of extra changes we cannot say. Whatever you have in the branch seems to be working although looking again it seem you bump the timeout to 11s so this seems most likely the reason then.