gh-pages stops getting updates

[ryandawsonuk]

Everything was logging fine but then mysteriously stopped. Tried:

• Removing gh-pages branch. I thought it would put it back but it doesn't so I then had to re-instate it.
• Restarting the jenkins-x-controllerbuild pod.

I want to try doing a jx upgrade boot but that gets stuck on issues with the cherry-picks.

[ryandawsonuk]

Managed to 'fix' it by doing a jx upgrade platform... is that the intended fix?

[marlovil]

Hi! Same problem here.

From time to time, JX stops putting logs to gh-pages. Last time, I performed a 'jx upgrade platform' that solved the issue...

[axsaucedo]

I can confirm that this is still an issue in:

NAME               VERSION
jx                 2.0.1029
Kubernetes cluster v1.14.8-gke.12
kubectl            v1.16.3
helm client        Client: v2.13.0+g79d0794
git                2.17.1
Operating System   Ubuntu 18.04.3 LTS

[axsaucedo]

We had this issue again - we couldn't just run jx upgrade platform nor jx upgrade boot as it was giving us merge conflicts, so we had to uninstall the cluster and do a full installation. This fixed it now. Next time this happens I want to try the following steps:

1. Remove the gh-pages branch
2. Change all storage components from jx-requirements to false (disabled) and wait for gitops update
3. Change all storage components from jx-requirements back to true
   The assumption is that this would re-create the gh-pages and the relevant triggers. If anyone else gets this issue again it would be great to hear if this fixes it for you.

[marlovil]

Today, again, the same problem.

But I've found the responsible piece that is in charge of doing this.

This is the pod: jenkins-x-controllerbuild-yyyyyyyyyy-zzzzz and these are the logs:

{"level":"info","msg":"storing logs for activity mycompany-jxqsapp-master-81 into storage at jenkins-x/logs/mycompany/jxqsapp/master/81.log","time":"2020-02-05T12:58:39Z"} {"level":"info","msg":"shallow cloning https://github.com/mycompany/jxqsapp.git branch gh-pages","time":"2020-02-05T12:58:39Z"} {"level":"warning","msg":"failed to shallow clone https://github.com/mycompany/jxqsapp.git branch gh-pages due to: failed to init a new git repository in directory /tmp/jenkins-x-collect632787736: git output: : failed to run 'git init' command in directory '/tmp/jenkins-x-collect632787736', output: ''","time":"2020-02-05T12:58:39Z"} {"level":"error","msg":"failed to store logs for activity mycompany-jxqsapp-master-81 into storage at jenkins-x/logs/mycompany/jxqsapp/master/81.log: failed to init a new git repository in directory /tmp/jenkins-x-collect369966487: git output: : failed to run 'git init' command in directory '/tmp/jenkins-x-collect369966487', output: ''","time":"2020-02-05T12:58:39Z"}

To solve the issue simply restart the pod. All pending logs are written immediately.

@ryandawsonuk @axsaucedo try this next time ;)

[axsaucedo]

@marlovil this worked like a charm - you saved us yet another tedious re-install! Thanks!

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[axsaucedo]

/remove-lifecycle rotten

[axsaucedo]

I can confirm this is also present in lighthouse, it seems the issue is with controllerbuild pod. Using the latest version as of 07/07/2020, it seems bouncing the controllerbuild pod can resolve this and it "catches up" on previous notifications. At present a quickfix is basically to have a presistent bounce of the controllerbuild container.

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[marlovil]

/remove-lifecycle stale

[jenkins-x-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle stale

[jenkins-x-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.
If this issue is safe to close now please do so with /close.
Provide feedback via https://jenkins-x.io/community.
/lifecycle rotten

[jenkins-x-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

[jenkins-x-bot]

@jenkins-x-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.
Provide feedback via https://jenkins-x.io/community.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the jenkins-x/lighthouse repository.