Unable to upgrade providers when using custom fetchconfig

[arno-pons]

What steps did you take and what happened:

When trying to perform an upgrade of already deployed providers, configured for airgapped environment (custom config map), I have the following error in each provider's controller logs:

capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-23T17:52:29.626242023+02:00 E0723 15:52:29.626100       1 controller.go:329] "Reconciler error" err="failed to get configuration for the InfrastructureProvider with name infrastructure-azure. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="5c096dda-b062-4d1b-9fbb-02ec7e27602f"

Here, controller AddonProvider seems to check configuration of an InfrastructureProvider, but I have same error for BootstrapProvider controller for example, and InfrastructureProvider is complaining about BootstrapProvider:

capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:25:39.452452000+02:00 E0724 06:25:39.452326       1 controller.go:329] "Reconciler error" err="failed to get configuration for the BootstrapProvider with name bootstrap-kubeadm. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="infrastructureprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="InfrastructureProvider" InfrastructureProvider="capz-system/infrastructure-azure" namespace="capz-system" name="infrastructure-azure" reconcileID="6bbf901c-baf7-412d-83b2-d3d90f37f9b2"

If i delete current provider version and apply new one, this provider is deployed without error, so the fetchConfig seems to be ok.

What did you expect to happen:

Upgrade of providers should happens no matter what kind of configuration is used (custom fetchconfig or "github manifest").

Anything else you would like to add:

Here is a full log about a controller reconciliation showing that it detects that version has changed but fails to perform upgrade:

capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.818619253+02:00 I0724 06:47:30.818454       1 genericprovider_controller.go:62] "Reconciling provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.818842756+02:00 I0724 06:47:30.818731       1 preflight_checks.go:58] "Performing preflight checks" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.819070959+02:00 I0724 06:47:30.819002       1 preflight_checks.go:199] "Preflight checks passed" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:30.819305962+02:00 I0724 06:47:30.819217       1 phases.go:240] "No configuration secret was specified" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.117517239+02:00 I0724 06:47:31.117378       1 phases.go:257] "Custom fetch configuration config map was provided" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.119239063+02:00 I0724 06:47:31.119171       1 phases.go:161] "Loading provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.122121203+02:00 I0724 06:47:31.122021       1 phases.go:420] "Fetching provider" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.419996575+02:00 I0724 06:47:31.419866       1 phases.go:475] "Version changes detected, updating existing components" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"
capi-providers-cluster-api-operator-7fc9485cd9-jmmdt manager 2024-07-24T08:47:31.627523882+02:00 E0724 06:47:31.627407       1 controller.go:329] "Reconciler error" err="failed to get configuration for the InfrastructureProvider with name infrastructure-azure. Please check the provider name and/or add configuration for new providers using the .clusterctl config file" controller="addonprovider" controllerGroup="operator.cluster.x-k8s.io" controllerKind="AddonProvider" AddonProvider="caaph-system/addon-helm" namespace="caaph-system" name="addon-helm" reconcileID="485e07af-e97f-4314-bd52-1db42de41262"

From my understanding issue come from the fact that known providers list remains the default one. The current controller is only aware of it's own provider custom configuration (see here) where it should have all providers custom infos to be able to check them here in this loop.

Environment:

• Cluster-api-operator version: 0.11.0
• Cluster-api version: 1.7.4
• Minikube/KIND version: N/A
• Kubernetes version: (use kubectl version): v1.28.7
• OS (e.g. from /etc/os-release): Ubuntu 22.04.4 LTS

/kind bug

[furkatgofurov7]

/triage accepted

[furkatgofurov7]

As discussed in the CAPI Operator office hours, we might want to reach out to CAPI Maintainers and discuss the possible solution.

[furkatgofurov7]

/assign @w21froster

[k8s-ci-robot]

@furkatgofurov7: GitHub didn't allow me to assign the following users: w21froster.

Note that only kubernetes-sigs members with read permissions, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time.
For more information please see the contributor guide

In response to this:

/assign @w21froster

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.