Support `OCIRepository` for Crossplane installation

[maximiliantech]

What would you like to be added:

Currently, we just support Flux' HelmRepository resource to install Crossplane, see

service-provider-crossplane/pkg/component/crossplane_component.go

Lines 66 to 75 in 7a77129

	repo := &sourcev1.HelmRepository{
	ObjectMeta: metav1.ObjectMeta{
	Name: strings.ToLower(ComponentNameCrossplane),
	Namespace: rcontext.TenantNamespace(ctx),
	},
	Spec: sourcev1.HelmRepositorySpec{
	URL: c.ChartSpec.Repository,
	Timeout: &metav1.Duration{Duration: 1 * time.Minute},
	},
	}

In the future we only want to support pulling Helm charts (in general all images, charts etc.) from an OCI registry. There we have to use OCIRepository resources to pull Helm charts.

With the task, we remove the implementation to use HelmRepository and from now on only support OCIRepository resources from Flux. The structure of ProviderConfig API needs to be rewritten in order to match our proposal.

Authentication to private OCI registry is something that needs be taken care of as well, as part of this issue.

Why is this needed:
In some environments where Crossplane might not be stored in Helm repositories but OCI repositories instead, it would come in handy that the service-provider-crossplane supports this out of the box. Important is that the ProviderConfig makes it configurable how the provider should behave.

[maximiliantech]

As discussed with @ValentinGerlach, there are still open questions:

• Do we want to switch to OCI repositories exclusively?
  • Unifies the pull secrets setup across our landscape
  • Less hassle with cleaning up orphaned resources when switching between source types
  • OCI is the only supported registry type in some environments

[maximiliantech]

We decided to use OCI repositories exclusively. We will start implementing this as part of this issue right away. I will update the description above accordingly.

[maximiliantech]

With #46 so far, the implementation work should be done.
Outstanding tasks to do:

• update README
• update samples
• test that implementation works

The last part is a bit more tricky since the upstream Crossplane Helm chart is only stored in a Helm repository instead of an OCI repository. So to verify, we should do a helm pull and helm push of Crossplane into a private registry to test that pulling the Crossplane Helm chart out of a private OCI repository (and with private, I mean that we can including testing the functionality from @rdksap see #45) works.

[maximiliantech]

While implementing, more questions for solving this task came up: We don't just assume that the Crossplane Helm chart is behind a private registry but also the docker images for the actual deployment. We therefore need a way to configure this accordingly. Maybe a rework of the ProviderConfig could help.

@n3rdc4ptn and I created this proposal how a ProviderConfig could look like:

apiVersion: crossplane.services.openmcp.cloud/v1alpha1
kind: ProviderConfig
metadata:
  name: default
spec:
  versions: # mapping of 
    - version: v2.0.2
      chartURL: "ghcr.io/openmcp-project/openmcp/charts/crossplane:v2.0.2" # oci:// prefix will added in code
      imageURL: "ghcr.io/openmcp-project/openmcp/images/crossplane:v2.0.2" # injected in HelmRelease values
    - version: v1.19.0
      chartURL: "ghcr.io/openmcp-project/openmcp/charts/crossplane:v1.19.0" # oci:// prefix will added in code
      imageURL: "ghcr.io/openmcp-project/openmcp/images/crossplane:v1.19.0" 
  availableProviders:
    - name: provider-kubernetes
      package: xpkg.upbound.io/upbound/provider-kubernetes
      versions:
        - v0.16.0
  imagePullSecrets:
    - name: registry-secret # Is it really a list or do we just want to provide a single secret with different auth servers in it?

Open questions

Multiple image pull secrets

As described in the YAML above: Is it really a list or do we just want to provide a single secret with different auth servers in it?
Because if Helm chart location, docker image location and Crossplane provider images differ from one another, how should we decide which pull secret goes where?

Sync registry-secret to tenant namespaces

To support pulling Helm charts from authenticated OCI registry endpoints, we need to sync the registry-secret image pull secret from namespace "openmcp-system" in all tenant namespaces (e.g. mcp--937b68c4-a176-89b2-be0c-1bc777c806d3) on the Platform cluster.

[n3rdc4ptn]

Multiple image pull secrets

I would say we limit the ProviderConfig to one PullSecret, as the HelmRelease/OCIRepository only supports one. If we allow a list and then pick the first of the list to be the secret for the OCIRepo it would end up in some kind if implicit behavior which is not really seeable by the operator.
I would favor a more explicit approach of just one pullSecret which will be then used for images and HelmChart OCIRepo.

Another solution would be to have multiple parameters in the ProviderConfig:

crossplaneHelmChartPullSecretRef:
  name:
crossplaneOCIImagePullSecrets:
- name: ...
- name: ....

[reshnm]

Don't overcomplicate the configuration for image pull secrets.
For the initial version I would completely remove the configuration of additional image pull secrets from the ProviderConfig and only read the list of image pull secrets from your own ServiceProvider resource.
Assume that the image pull secrets are being used for both the helm chart and the container images.

[rdksap]

Interesting, I had the (maybe wrong) understanding that k8s can handle a list of imagePullSecrets (on pods or serviceAccounts) and would just iterate over them until a set of credentials works (or dies if none) 🤔

[reshnm]

Interesting, I had the (maybe wrong) understanding that k8s can handle a list of imagePullSecrets (on pods or serviceAccounts) and would just iterate over them until a set of credentials works (or dies if none) 🤔

That is correct, that is how it works. We don't need to know what image pull secret can be used for which location. We just pass all.

Also, in reality (means 99%) of use case we will only have a singe image pull secret because we store all our oci images within a single registry.

[rdksap]

Interesting, I had the (maybe wrong) understanding that k8s can handle a list of imagePullSecrets (on pods or serviceAccounts) and would just iterate over them until a set of credentials works (or dies if none) 🤔

That is correct, that is how it works. We don't need to know what image pull secret can be used for which location. We just pass all.

Also, in reality (means 99%) of use case we will only have a singe image pull secret because we store all our oci images within a single registry.

Got it. I guess the interesting scenario for multiple secrets is when one secret is about to expire and you want to create a new one next to it that will take over once the old one has expired.

[reshnm]

I see, that a FluxCD OCIRepository for flux only supports one secret.
Therefore we would need to know which of the secrets shall be used for the helm chart.

Also, I would not repeat chartURL and imageURL for each version. That looks redundant to me.
This could be simplified in my opinion:

spec:
  chart:
    oci:
      url: "ghcr.io/openmcp-project/openmcp/charts/crossplane"
      secretRef:
        name: ghcr
   image:
     url: "ghcr.io/openmcp-project/openmcp/images/crossplane"
     secretRef:
       name: xyz
  availableVersions:
  - v2.0.2
  - v1.19.0

Your ServiceProvider resource would look like this:

apiVersion: openmcp.cloud/v1alpha1
kind: ServiceProvider
metadata:
  name: "crossplane"
spec:
  image: "ghcr.io/openmcp-project/openmcp/images/service-provider-crossplane:v1.2.3"
  imagePullSecrets:
    - name: "ghcr"
    - name: "xyz"