Pod logs stop being pulled when container log files are rotated

[oliverf1]

As a followup of https://github.com/ansible/awx/issues/10366.\

Receptor is pulling logs from the pods using the kubernetes API log call

receptor/pkg/workceptor/kubernetes.go

Lines 273 to 276 in 156e6e2

	logreq := kw.clientset.CoreV1().Pods(kw.pod.ObjectMeta.Namespace).GetLogs(kw.pod.Name, &corev1.PodLogOptions{
	Container: "worker",
	Follow: true,
	})

.
However as explained in kubernetes/kubernetes#59902, if the container log files (on the node) are rotated the log stream stops. As a consequence logs are not fully sent to AWX database. In such case, the AWX jobs will be flagged as failed, even though it went to the end, because the whole logs are not available.

[shanemcd]

Related: ansible/awx#11338

[shanemcd]

Trying to fix this here: #683

[domq]

Trying to fix this here: #683

I seem to still get only the first chunk of logs. Running AWX receptor version 1.4.1 against OpenShift 3.11's (I know, I know) Kubernetes 1.11.0+d4cacc0. (Edit: this appears to not be a timing issue, as I initially thought.)

It is worth noting that the /tmp/receptor/awx-0/*/stdout files do not end with (or contain) an { "eof": true } marker in this case, indicating that Receptor at least understands that it has work remaining. If that latter fact means that I should open another issue, kindly let me know.

[Klaas-]

Crosslinking ansible/awx#14158 I am guessing this may have the same root cause?

[Klaas-]

and upstream has a PR for something that sounds related: kubernetes/kubernetes#118500

[Klaas-]

A potential fix was merged in upstream kubernetes in kubernetes/kubernetes#115702

[Klaas-]

I updated kubernetes to 1.29 -- this seems to fix the issues for me