[self-hosted] 20022.08.0 support bundle does not contain pod logs

[adrienthebo]

Bug description

In Gitpod 2022.07.x, support bundles contained logs and log-collector directories containing logs of running pods and prior pods, respectively. Recent troubleshooting has shown that support bundles created from Gitpod 2022.08.0 don't contain these directories.

Steps to reproduce

Create a support bundle based on 2022.08.0 and confirm that they are missing the expected log directories.

Workspace affected

No response

Expected behavior

No response

Example repository

No response

Anything else?

No response

Front conversations

[mrsimonemms]

My anecdotal evidence is that the Fluent Bit logs are missing on some installations and not on others, regardless of the version of Gitpod. When I developed it, I believe I tried it on k3s, GKE and AKS but I know of at least one GKE user that's not got these appearing in their support bundles in 2022.7.x.

Will investigate

[adrienthebo]

I performed some preliminary testing yesterday using a dev license on AKS; the support bundle did contain logs as expected. It might be that this is a limitation in how we're sending logs via replicated for production licenses or a mis-firing redactor.

Edit: Good catch regarding the absent logs in 2022.7.x; I suspected it was a regression but if these aren't present on 2022.8.x then this scenario is ruled out.

[adrienthebo]

I've rooted around the support-bundle log collector and kotsadm interface to support-bundle to look for obvious pitfalls or where more troubleshooting information might lie. The log collection code is a little bit strange and warrants a closer look; if logs can't be collected then logs should be stored in the support bundle but I'm not certain all of the error cases are checked.

On potential concern might be that the kotsadm service account might not have permissions to read logs from the Gitpod pods, but that's conjecture.

Speaking more broadly, this has now affected multiple customers and is more than a one-off error.

[jimmybrancaccio]

So I just got a support bundle from someone and they're using release-2022.08.0.10. It included both logs and log-collector directories with logs in each. (Front ticket).

[mrsimonemms]

The support bundle in the first message has a log-collector directory, but the files are called log-collector-errors.json and the contents are about how it cannot create the /gitpod directory on the node.

I can see that it's on GKE. There's definitely at least one other customer who's having issues on GKE, so I'm going to focus efforts on that

[mrsimonemms]

Having done some investigation, it looks like it's due to the /gitpod/log-collector directory is not always present on the nodes. I've done a fix in #13233 which ensures that directory is always present and that stops filling the support bundles up with all the log-collector-errors.json files.