[Flaking Test][sig-node] Container Runtime blackbox test on terminated container should report termination message from file when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance]

[stmcginnis]

Which jobs are flaking?

master-blocking

• gce-ubuntu-master-containerd

Which tests are flaking?

E2eNode Suite.[It] [sig-node] Container Runtime blackbox test on terminated container should report termination message from file when pod succeeds and TerminationMessagePolicy FallbackToLogsOnError is set [NodeConformance] [Conformance]

Prow
Triage

Since when has it been flaking?

1/10/2025, 6:57:34 PM
1/21/2025, 5:54:07 AM

Testgrid link

https://testgrid.k8s.io/sig-release-master-blocking#ci-crio-cgroupv1-node-e2e-conformance

Reason for failure (if possible)

{ failed [FAILED] Timed out after 300.001s.
Expected
    <v1.PodPhase>: Failed
to equal
    <v1.PodPhase>: Succeeded
In [It] at: k8s.io/kubernetes/test/e2e/common/node/runtime.go:157 @ 01/21/25 12:18:04.935
}

Anything else we need to know?

N/A

Relevant SIG(s)

/sig node
cc @kubernetes/release-team-release-signal

[SergeyKanzhelev]

CI meeting notes:

• Very rare flake.
• We cannot find containerd job for this test to compare.

/cc @haircommander
/triage accepted

[haircommander]

@bitoku would you PTAL?

[bitoku]

certainly.

/assign

[bitoku]

It seems like the container failed with exitCode 255.
I'm not sure why but it might have failed parsing exit file.
cri-o/cri-o#8937 this change might give an information when it happens again.

[wendy-ha18]

Hi @bitoku, thanks for the PR cri-o/cri-o#8937. I just checked Triage again today and saw that after your PR being merged, at the same day this test failed again with Prow here: 22/01/2025, 15:24:08
{ failed [FAILED] checking for ready nodes: Not ready nodes: ", ip-172-31-1-25.ec2.internal" In [DeferCleanup (Each)] at: k8s.io/kubernetes/test/e2e/framework/node/init/init.go:35 @ 01/22/25 05:31:28.337 }.

Triage link. Not sure if it helps your investigation but just want to let you know!

[Rajalakshmi-Girish]

@bitoku Can you please tell whether this issue will block the v1.33.0-alpha.1 cut, which is scheduled for Tuesday, 4th February UTC?

[bitoku]

@Rajalakshmi-Girish I'm not yet sure but I saw some suspicious logs in cri-o so I think it's a bug in cri-o.
Also this is a rare flaky test, so I don't think it's a blocker for the new release.

[bitoku]

There's likely to be a bug in cri-o side. We updated the CI to use the updated version of cri-o, which should output more logs when the issue happens.
Once we can get another occurrence, I can continue investigation.

[stmcginnis]

This appears to be passing now. Will wait on confirmation to close.

[SergeyKanzhelev]

This seems to be flaking now: https://storage.googleapis.com/k8s-triage/index.html?test=Container%20Runtime%20blackbox%20test%20on%20terminated%20container%20should%20report%20termination%20message%20from%20file%20when%20pod%20succeeds%20and%20TerminationMessagePolicy%20FallbackToLogsOnError%20is%20set

very rare flakes, setting lower priority. Also flaking on containerd as well, so not a cri-o issue (it seems)

/priority important-longterm

[esotsal]

/cc

[wendy-ha18]

Hi folks, thanks a lot for your support and attention on this issue.
The release cycle for v1.34 will start soon, and since this is still open, I will carry it over to the latest milestone.

/milestone v1.34