Branch protector is failing with "update kubernetes-sigs: update metrics-server: update gh-pages"

[chaodaiG]

What happened:

Branch protector job is failing consistently:

{"component":"branchprotector","file":"k8s.io/test-infra/prow/cmd/branchprotector/protect.go:118","func":"main.(*Errors).add","level":"info","msg":"update kubernetes-sigs: update metrics-server: update gh-pages from protected=false: get policy: required prow jobs require branch protection","severity":"info","time":"2022-04-20T20:00:31Z"}
{"component":"branchprotector","error":"update kubernetes-sigs: update metrics-server: update gh-pages from protected=false: get policy: required prow jobs require branch protection","file":"k8s.io/test-infra/prow/cmd/branchprotector/protect.go:160","func":"main.main","level":"error","msg":"0","severity":"error","time":"2022-04-20T20:00:31Z"}
{"component":"branchprotector","file":"k8s.io/test-infra/prow/cmd/branchprotector/protect.go:162","func":"main.main","level":"fatal","msg":"Encountered 1 errors protecting branches","severity":"fatal","time":"2022-04-20T20:00:31Z"}

link: https://prow.k8s.io/view/gs/kubernetes-jenkins/logs/ci-test-infra-branchprotector/1516867945198784512
see more: https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-test-infra-branchprotector

What you expected to happen:

it should pass

/sig testing

[BenTheElder]

this error message is a bit inscrutable, does anyone know what actually happened here?
/area prow
cc @cjwagner @stevekuznetsov

[cjwagner]

It sounds like this configuration is invalid:

test-infra/config/prow/config.yaml

Lines 562 to 565 in a0fe9c2

	metrics-server:
	branches:
	gh-pages:
	protect: false

The gh-pages branch disables branch protection, but since required ProwJobs are configured for the branch that is not allowed. I'm not sure why this isn't validated earlier, seems like we should be able to do this validation in checkconfig.
#24222 @stevehipwell

[stevehipwell]

@Raffo it looks like the documentation PR that was merged has broken this. We might need it rolled back until the implications have been considered.

[stevehipwell]

@cjwagner additional automation has been added to the repo since branch protection was disabled. The branch protection needs to stay off for the Helm chart publishing, so the new workflow needs reviewing with this information.

[Raffo]

@stevehipwell wow, how can a failure in the pages for external-dns break test-infra is unknown to me. Working on a rollback now.

[Raffo]

Revert PR: kubernetes-sigs/external-dns#2727

[Raffo]

The revert PR has been merged, I hope this fixes the problem for now.

@stevehipwell let's chat for what's needed to reintroduce it safely.

[BenTheElder]

Still failing. Need to revert to test-infra config next?

[stevehipwell]

@cjwagner sorry I've been away for a long weekend. Did you figure out if the issue was the ExternalDNS config or the Metrics Server config? I'm not aware of any recent changes to Metrics Server CI which would have this suddenly failing if it was validated when I opened the PR, but maybe it didn't validate fully? ExternalDNS did have a CI change but I don't see any enforced prow jobs so I'm not sure if the code @Raffo reverted actually needed to be?

[Raffo]

@stevehipwell yeah I only reverted to make sure we can make progress on this, happy to reopen the PR again to external-dns once you tell me we're good to go.

[stevehipwell]

@Raffo I think it'd be OK to open it back up again but I think @cjwagner is in a better position to confirm if its safe to do so or not.

[cjwagner]

The config fix to remove the required jobs from kubernetes-sigs/metrics-server:gh-pages (#26173) seems to have been sufficient. The branchprotector is passing reliably again: https://prow.k8s.io/job-history/gs/kubernetes-jenkins/logs/ci-test-infra-branchprotector
I don't see any indication that the external-dns repo was involved at all here. Unless I'm missing something that change didn't need to be reverted and can be safely reapplied. Please keep an eye on the branchprotector status after reintroducing the change just in case though.

[cjwagner]

fixed by #26173
/close

[k8s-ci-robot]

@cjwagner: Closing this issue.

In response to this:

fixed by #26173
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.