[Bug]: Pipeline issues - AKS cluster (and other Azure resources) not created reliably

[drew0ps]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

containerservice.azure.upbound.io/v1beta1 - KubernetesCluster

Resource MRs required to reproduce the bug

No response

Steps to Reproduce

• Run a pipeline targeting examples/containerservice/v1beta1/kubernetescluster.yaml
• Observe Kubernetes cluster creation

What happened?

This run is using the examples/containerservice/v1beta1/kubernetescluster.yaml - same as main.
https://github.com/crossplane-contrib/provider-upjet-azure/actions/runs/10682720219

Relevant Error Output Snippet

No response

Crossplane Version

main, whatever is used by the uptests pipeline

Provider Version

main

Kubernetes Version

No response

Kubernetes Distribution

No response

Additional Info

I have been trying to merge in my PR but the uptests keep failing, even though manual tests succeed with the newly added resource. Because the resource relies on an AKS cluster present in the cloud I ran the pipeline with AKS defined in the CR and noticed instability, several unsuccesful runs where the cluster was stuck in Creating state throughout the 30 minute run.

[drew0ps]

I ran another test against /test-examples="examples/containerservice/v1beta1/kubernetescluster.yaml" : https://github.com/crossplane-contrib/provider-upjet-azure/actions/runs/10698538178/job/29658186863
The cluster was in creating state for 20 minutes after which the pipeline failed.

[turkenf]

Hi @drew0ps, thank you for raising the issue.

I think this issue is not only for pipeline/uptest. I just tried in a local kind cluster and the resource cannot be created for over 20 minutes without any errors.

• k get managed

NAME                                                         SYNCED   READY   EXTERNAL-NAME                 AGE
resourcegroup.azure.upbound.io/example-containerservice-ft   True     True    example-containerservice-ft   23m

NAME                                                              SYNCED   READY   EXTERNAL-NAME   AGE
kubernetescluster.containerservice.azure.upbound.io/example-xft   True     False   example-xft     24m

status:
  atProvider: {}
  conditions:
  - lastTransitionTime: "2024-09-04T09:07:18Z"
    reason: ReconcileSuccess
    status: "True"
    type: Synced
  - lastTransitionTime: "2024-09-04T09:07:18Z"
    reason: Creating
    status: "False"
    type: Ready
  - lastTransitionTime: "2024-09-04T09:07:31Z"
    reason: Success
    status: "True"
    type: LastAsyncOperation

• and the logs:

2024-09-04T11:34:01+02:00	DEBUG	provider-azure	Diff detected	{"uid": "a62874ca-60aa-4b97-a882-e2ff50b2ec75", "name": "example-xft", "gvk": "containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"api_server_access_profile.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_access_profile.0.authorized_ip_ranges.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_access_profile.0.authorized_ip_ranges.2201616582\":*terraform.ResourceAttrDiff{Old:\"\", New:\"192.168.1.0/24\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_access_profile.0.vnet_integration_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_authorized_ip_ranges.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"auto_scaler_profile.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"current_kubernetes_version\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.kubelet_disk_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.max_pods\":*terraform.ResourceAttrDiff{Old:\"\", New:\"0\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"default\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"default_node_pool.0.node_count\":*terraform.ResourceAttrDiff{Old:\"\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.node_labels.%\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.orchestrator_version\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.os_disk_size_gb\":*terraform.ResourceAttrDiff{Old:\"\", New:\"0\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.os_disk_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Managed\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.os_sku\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.scale_down_mode\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Delete\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"VirtualMachineScaleSets\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"default_node_pool.0.ultra_ssd_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.vm_size\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Standard_D2_v2\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.workload_runtime\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"dns_prefix\":*terraform.ResourceAttrDiff{Old:\"\", New:\"exampleaks1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"fqdn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"http_application_routing_zone_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.0.principal_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.0.tenant_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.0.type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"SystemAssigned\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"image_cleaner_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"image_cleaner_interval_hours\":*terraform.ResourceAttrDiff{Old:\"\", New:\"48\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kube_admin_config.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kube_admin_config_raw\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:true, Type:0x0}, \"kube_config.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kube_config_raw\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:true, Type:0x0}, \"kubelet_identity.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kubernetes_version\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"location\":*terraform.ResourceAttrDiff{Old:\"\", New:\"westeurope\", NewComputed:false, NewRemoved:false, NewExtra:\"West Europe\", RequiresNew:true, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example-xft\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"network_profile.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"node_resource_group\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"node_resource_group_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"oidc_issuer_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"portal_fqdn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"private_cluster_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"private_cluster_public_fqdn_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"private_dns_zone_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"private_fqdn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"public_network_access_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"resource_group_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example-containerservice-ft\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"role_based_access_control_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"run_command_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"sku_tier\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Free\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"support_plan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"KubernetesOfficial\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.%\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.Environment\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Production\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"windows_profile.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"workload_identity_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-09-04T11:34:01+02:00	DEBUG	provider-azure	Waiting for external resource existence to be confirmed	{"controller": "managed/containerservice.azure.upbound.io/v1beta1, kind=kubernetescluster", "request": {"name":"example-xft"}, "uid": "a62874ca-60aa-4b97-a882-e2ff50b2ec75", "version": "5356", "external-name": "example-xft"}
2024-09-04T11:34:09+02:00	DEBUG	provider-azure	Reconciling	{"controller": "managed/containerservice.azure.upbound.io/v1beta1, kind=kubernetescluster", "request": {"name":"example-xft"}}
2024-09-04T11:34:09+02:00	DEBUG	provider-azure	Connecting to the service provider	{"uid": "a62874ca-60aa-4b97-a882-e2ff50b2ec75", "name": "example-xft", "gvk": "containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster"}
2024-09-04T11:34:10+02:00	DEBUG	provider-azure	Instance state not found in cache, reconstructing...	{"uid": "a62874ca-60aa-4b97-a882-e2ff50b2ec75", "name": "example-xft", "gvk": "containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster"}
2024-09-04T11:34:10+02:00	DEBUG	provider-azure	Observing the external resource	{"uid": "a62874ca-60aa-4b97-a882-e2ff50b2ec75", "name": "example-xft", "gvk": "containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster"}
2024-09-04T11:34:11+02:00	DEBUG	provider-azure	Diff detected	{"uid": "a62874ca-60aa-4b97-a882-e2ff50b2ec75", "name": "example-xft", "gvk": "containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"api_server_access_profile.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_access_profile.0.authorized_ip_ranges.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_access_profile.0.authorized_ip_ranges.2201616582\":*terraform.ResourceAttrDiff{Old:\"\", New:\"192.168.1.0/24\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_access_profile.0.vnet_integration_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"api_server_authorized_ip_ranges.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"auto_scaler_profile.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"current_kubernetes_version\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.kubelet_disk_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.max_pods\":*terraform.ResourceAttrDiff{Old:\"\", New:\"0\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"default\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"default_node_pool.0.node_count\":*terraform.ResourceAttrDiff{Old:\"\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.node_labels.%\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.orchestrator_version\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.os_disk_size_gb\":*terraform.ResourceAttrDiff{Old:\"\", New:\"0\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.os_disk_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Managed\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.os_sku\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.scale_down_mode\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Delete\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"VirtualMachineScaleSets\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"default_node_pool.0.ultra_ssd_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.vm_size\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Standard_D2_v2\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"default_node_pool.0.workload_runtime\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"dns_prefix\":*terraform.ResourceAttrDiff{Old:\"\", New:\"exampleaks1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"fqdn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"http_application_routing_zone_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.#\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.0.principal_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.0.tenant_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"identity.0.type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"SystemAssigned\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"image_cleaner_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"image_cleaner_interval_hours\":*terraform.ResourceAttrDiff{Old:\"\", New:\"48\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kube_admin_config.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kube_admin_config_raw\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:true, Type:0x0}, \"kube_config.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kube_config_raw\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:true, Type:0x0}, \"kubelet_identity.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"kubernetes_version\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"location\":*terraform.ResourceAttrDiff{Old:\"\", New:\"westeurope\", NewComputed:false, NewRemoved:false, NewExtra:\"West Europe\", RequiresNew:true, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example-xft\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"network_profile.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"node_resource_group\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"node_resource_group_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"oidc_issuer_url\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"portal_fqdn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"private_cluster_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"private_cluster_public_fqdn_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"private_dns_zone_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"private_fqdn\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"public_network_access_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"resource_group_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"example-containerservice-ft\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"role_based_access_control_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"run_command_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"true\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"sku_tier\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Free\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"support_plan\":*terraform.ResourceAttrDiff{Old:\"\", New:\"KubernetesOfficial\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.%\":*terraform.ResourceAttrDiff{Old:\"0\", New:\"1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"tags.Environment\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Production\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"windows_profile.#\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"workload_identity_enabled\":*terraform.ResourceAttrDiff{Old:\"\", New:\"false\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-09-04T12:11:13+02:00	DEBUG	provider-azure	Async create ended.	{"trackerUID": "0984b955-480c-4ef3-84dc-1287cc5c1b26", "resourceName": "example-ftest", "gvk": "containerservice.azure.upbound.io/v1beta1, Kind=KubernetesCluster", "error": null, "tfID": "/subscriptions/038f2b7c-3265-43b8-8624-c9ad5da610a8/resourceGroups/example-ftest/providers/Microsoft.ContainerService/managedClusters/example-ftest"}

[drew0ps]

Hi @turkenf, thank you for the response and checking.

I suspect there is something with this specific CR since I also checked locally with the monolith (make run) and there I succeeded with a different CR (my k8s extensions story), although I run rancher desktop:

apiVersion: kubernetesconfiguration.azure.upbound.io/v1beta1
kind: KubernetesClusterExtension
metadata:
  annotations:
    meta.upbound.io/example-id: kubernetesconfiguration/v1beta1/kubernetesclusterextension
  labels:
    testing.upbound.io/example-name: example
  name: example
spec:
  forProvider:
    clusterId: subscriptions/mysub/resourceGroups/example/providers/Microsoft.ContainerService/managedClusters/example
    extensionType: microsoft.flux
    releaseTrain: Stable
    configurationSettings:
      retentionPolicy: "1"
  providerConfigRef:
    name: default

---

apiVersion: containerservice.azure.upbound.io/v1beta1
kind: KubernetesCluster
metadata:
  annotations:
    meta.upbound.io/example-id: kubernetesconfiguration/v1beta1/kubernetesclusterextension
  labels:
    testing.upbound.io/example-name: example
  name: example
spec:
  forProvider:
    defaultNodePool:
    - name: default
      nodeCount: 1
      vmSize: Standard_D4ds_v5
    dnsPrefix: example-aks
    identity:
    - type: SystemAssigned
    location: North Europe
    resourceGroupNameSelector:
      matchLabels:
        testing.upbound.io/example-name: example

---

apiVersion: azure.upbound.io/v1beta1
kind: ResourceGroup
metadata:
  annotations:
    meta.upbound.io/example-id: kubernetesconfiguration/v1beta1/kubernetesclusterextension
  labels:
    testing.upbound.io/example-name: example
  name: example
spec:
  forProvider:
    location: North Europe

k get kubernetesclusterextension.kubernetesconfiguration.azure.upbound.io/example  kubernetescluster.containerservice.azure.upbound.io/example resourcegroup.azure.upbound.io/example
NAME                                                                          SYNCED   READY   EXTERNAL-NAME   AGE
kubernetesclusterextension.kubernetesconfiguration.azure.upbound.io/example   True     True    example         17m

NAME                                                          SYNCED   READY   EXTERNAL-NAME   AGE
kubernetescluster.containerservice.azure.upbound.io/example   True     True    example         17m

NAME                                     SYNCED   READY   EXTERNAL-NAME   AGE
resourcegroup.azure.upbound.io/example   True     True    example         17m

Conditions:
    Last Transition Time:  2024-09-04T09:08:20Z
    Reason:                ReconcileSuccess
    Status:                True
    Type:                  Synced
    Last Transition Time:  2024-09-04T09:12:41Z
    Reason:                Available
    Status:                True
    Type:                  Ready
    Last Transition Time:  2024-09-04T09:12:13Z
    Reason:                Success
    Status:                True
    Type:                  LastAsyncOperation

The differences are:
Location: Its north europe on mine and west europe on the non-working one
vmsize is different
The following options which are missing from my CR:
tags:
Environment: Production
apiServerAccessProfile:
- authorizedIpRanges:
- 192.168.1.0/24

In this run the AKS cluster creation succeeded: https://github.com/crossplane-contrib/provider-upjet-azure/actions/runs/10699008353/job/29659653711
Target: /test-examples="examples/kubernetesconfiguration/v1beta1/kubernetesclusterextension.yaml"

[turkenf]

I am experiencing the same issue in the examples you gave, and this may be caused by our subscriptions 🤔

[drew0ps]

Hm that could be, but that does not explain why the AKS cluster succeeds with the creation on this run on subscription "2895a7df-ae9f-41b8-9e78-3ce4926df838" :

    logger.go:42: 09:59:01 | case/0-apply |     conditions:
    logger.go:42: 09:59:01 | case/0-apply |     - lastTransitionTime: "2024-09-04T09:44:10Z"
    logger.go:42: 09:59:01 | case/0-apply |       reason: Available
    logger.go:42: 09:59:01 | case/0-apply |       status: "True"
    logger.go:42: 09:59:01 | case/0-apply |       type: Ready
    logger.go:42: 09:59:01 | case/0-apply |     - lastTransitionTime: "2024-09-04T09:39:14Z"
    logger.go:42: 09:59:01 | case/0-apply |       reason: ReconcileSuccess
    logger.go:42: 09:59:01 | case/0-apply |       status: "True"
    logger.go:42: 09:59:01 | case/0-apply |       type: Synced
    logger.go:42: 09:59:01 | case/0-apply |     - lastTransitionTime: "2024-09-04T09:39:14Z"
    logger.go:42: 09:59:01 | case/0-apply |       reason: Success
    logger.go:42: 09:59:01 | case/0-apply |       status: "True"
    logger.go:42: 09:59:01 | case/0-apply |       type: LastAsyncOperation

This succeeds after like 5 minutes. The extension install still fails though, which eventually blocks my PR. All of this works on my own Azure env though.

[drew0ps]

Quick update here - the AKS cluster does not install because there is no free tier AKS anymore in West Europe. In North Europe it is still failing but with a different reason most probably. I could reproduce it with make e2e though and there I get the following event in the Azure Activity Log as I mentioned in my PR:

The PutAgentPoolHandler.PUT request limit has been exceeded for agentpool /subscriptions/mysub/resourcegroups/example-rg-kce/providers/Microsoft.ContainerService/managedClusters/example-kc-kce/agentPools/default, please retry again in 36 seconds

The same message comes up all the time so retry doesn't help.

[drew0ps]

For some reason the AKS cluster creation enters an update loop trying to remove a field called maxSurge. However, I found that if the field is added to the manifests this doesn't happen. It still seems to be a bug, although there is a workaround:

spec:
  forProvider:
    defaultNodePool:
    - name: default
      nodeCount: 1
      vmSize: Standard_D2_v2
      upgradeSettings:
      - maxSurge: '10%'
...