SURE-6622: Kubewarden Policy Server 1.6.0 can not pull policy artefacts from harbor registry when authentication is required

[jhmarina]

Issue description:

We received this case from one of our SUSE colleagues: Martin Weiss.  He reported that Kubewarden Policy Server 1.6.0 could not pull policy OCI artefacts from harbor registry when authentication is required.

Business impact:

The policy server does not work with authentication against harbor to fetch the policies.

Repro steps:

Replicate kubewarden policies to harbor registry
Ensure the project requires authentication
Configure the policy server to fetch the policies with authentication from the registry

This setup is based on this example:

https://github.com/Martin-Weiss/rancher-fleet/tree/main/kubewarden/defaults
https://github.com/Martin-Weiss/rancher-fleet/tree/main/kubewarden
 

Workaround:

Is workararound available and implemented? NO

Actual behavior:

Policy pull fails

Expected behavior:

Policy pull works

Additional notes:

[flavio]

I tried to reproduce the issue, but I wasn't able. Everything worked as expected.

These are the details of my setup:

• Harbor 2.8.0 deployed on a dedicated VM, using a certificate signed by a self-signed CA
• Kubewarden 1.6.0 running inside of another VM (via minikube)

I created a private project on Harbor and I pushed a Kubewarden policy inside of it using kwctl. kwctl was able to push and pull the policy after I did a docker login.

I created a Kubernetes Secret containing the authentication details of my Harbor server (as described here):

kubectl --namespace kubewarden create secret docker-registry secret-harbor
   --docker-username=admin
   --docker-password=Harbor12345
  --docker-server=harbor.svc.lan

I changed the default Policy Server object using kubectl edit, I added  the following lines under the spec:

spec:
  imagePullSecret: secret-harbor
     sourceAuthorities:
        harbor.svc.lan:
        - |
          --BEGIN CERTIFICATE--
          My self-signed CA cert
          --END CERTIFICATE--

I waited for the policy server to be redeployed, then I created a Kubewarden policy consuming the policy hosted by Harbor.

I also made other tests, like removing the imagePullSecret value to ensure the policy could not be downloaded. The test worked, with Policy Server failing to boot because it wasn't authorized to download the policy.

[flavio]

@Martin-Weiss: feel free to provide the details about your environment here

[flavio]

Conversation around this topic can also be found here https://suse.slack.com/archives/C050E1NKTCJ/p1688634061313089

@jhmarina this is on a private slack instance that is not accessible by members of the community. I've looked into that and this is about a different topic: Rancher Kubewarden UI not being accessible under air-gapped environments

[flavio]

I accidentally closed the issue, sorry.

I'm moving it to blocked, waiting for more information about how to reproduce the issue.

• Which version of Harbor is being used?
• How is it deployed?
• What kind of tls protocols are supported by whoever does the termination? We still have kwctl fails to pull policy in case authentication and authorization are used on the registry and registry auth source does not support tls 1.3 kwctl#361 open, which is about not being able to pull from a registry that does not support tls 1.3
• Can you pull and push policies to the Harbor instance after docker login is done?

[jhmarina]

Conversation around this topic can also be found here https://suse.slack.com/archives/C050E1NKTCJ/p1688634061313089

@jhmarina this is on a private slack instance that is not accessible by members of the community. I've looked into that and this is about a different topic: Rancher Kubewarden UI not being accessible under air-gapped environments

I removed the link to the conversation. I copied it from the original JIRA issue, apologies for not double-checking!

[Martin-Weiss]

FYI - seems this problem is solved and it had been a docker credentials set that was "wrong" in the registry secret.
-> do you see any chance to improve the logging output so we get some more info on what is causing a failed pull?

[flavio]

Great, thanks for the update