[Bug]: No 'LateInitialize' management policy combined with an ungraceful provider pod termination can lead to attempted resource recreation

[b-deam]

Is there an existing issue for this?

• I have searched the existing issues

Affected Resource(s)

• authorization.azure.upbound.io/v1beta1 - RoleAssignment
• managedidentity.azure.upbound.io/v1beta1 - UserAssignedIdentity

Resource MRs required to reproduce the bug

apiVersion: authorization.azure.upbound.io/v1beta1
kind: RoleAssignment
metadata:
  name: my-test
spec:
  deletionPolicy: Delete
  forProvider:
    name: <GUID>
    principalId: <PRINCIPALID>
    principalType: ServicePrincipal
    roleDefinitionName: Contributor
    scope: /subscriptions/<SUB_ID>
  initProvider: {}
  managementPolicies:
  - Create
  - Update
  - Delete
  - Observe
  providerConfigRef:
    name: azure-provider-config
---
apiVersion: managedidentity.azure.upbound.io/v1beta1
kind: UserAssignedIdentity
metadata:
  name: user-assigned-identity-test
spec:
  deletionPolicy: Delete
  forProvider:
    location: eastus2
    name: user-assigned-identity-test
    resourceGroupName: <RG_NAME>
  initProvider: {}
  managementPolicies:
  - Create
  - Update
  - Delete
  - Observe
  providerConfigRef:
    name: azure-provider-config

Steps to Reproduce

1. Forcefully delete the respective provider pod(s), e.g. kubectl delete pod/provider-azure-authorization-ca718ab078b2-9c7fcdd55-hwrfn -n crossplane --force --grace-period=0 , this may take a few tries
2. Observe that the provider attempts to recreate the resource

What happened?

Resources should not be attempted to be created, rather if a field changes that requires resource recreation we should see an async update failure similar to:

NAME                                                      SYNCED   READY   STATUS

UserAssignedIdentity/my-user-assigned-id                             False    True    ReconcileError: update failed: async update
failed: refuse to update the external resource because the following update requires replacing it: cannot change the value of the argument "resource_group_name" from "app-1" to ""

Relevant Error Output Snippet

Crossplane beta trace for RoleAssignment:

NAME                     RESOURCE   SYNCED   READY   STATUS
RoleAssignment/my-test              False    False   ReconcileError: create failed: async create failed: failed to create the resource: [{0 A resource with the ID "/subscriptions/<subscription_id>/providers/Microsoft.Authorization/roleAssignments/11f7c29f-df3c-4b4f-99cb-a41e2f951bec" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for "azurerm_ro
le_assignment" for more information.  []}]

Logs from provider-azure-authorization after force deleting pod:

2024-08-28T04:35:11Z	DEBUG	provider-azure	Starting	{"sync-interval": "1h0m0s", "poll-interval": "10m0s", "poll-jitter": "30s", "max-reconcile-rate": 100}
2024-08-28T04:35:14Z	INFO	provider-azure	Beta feature enabled	{"flag": "EnableBetaManagementPolicies"}
2024-08-28T04:35:14Z	DEBUG	provider-azure	Calling the inner handler for Create event.	{"gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "name": "my-test", "queueLength": 0}
2024-08-28T04:35:14Z	DEBUG	provider-azure	Reconciling	{"controller": "managed/authorization.azure.upbound.io/v1beta1, kind=roleassignment", "request": {"name":"my-test"}}
2024-08-28T04:35:14Z	DEBUG	provider-azure	Connecting to the service provider	{"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Instance state not found in cache, reconstructing...	{"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Observing the external resource	{"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Diff detected	{"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"11f7c29f-df3c-4b4f-99cb-a41e2f951bec\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"principal_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"c2c23f94-189a-4a64-b46a-4eb52739ea49\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"principal_type\":*terraform.ResourceAttrDiff{Old:\"\", New:\"ServicePrincipal\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"role_definition_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"Contributor\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"scope\":*terraform.ResourceAttrDiff{Old:\"\", New:\"/subscriptions/<SUB_ID>\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"skip_service_principal_aad_check\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Async create starting...	{"trackerUID": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "resourceName": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "tfID": ""}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Creating the external resource	{"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Successfully requested creation of external resource	{"controller": "managed/authorization.azure.upbound.io/v1beta1, kind=roleassignment", "request": {"name":"my-test"}, "uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "version": "572125", "external-name": "", "external-name": ""}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Calling the inner handler for Update event.	{"gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "name": "my-test", "queueLength": 0}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Reconciling	{"controller": "managed/authorization.azure.upbound.io/v1beta1, kind=roleassignment", "request": {"name":"my-test"}}
2024-08-28T04:35:18Z	DEBUG	provider-azure	Connecting to the service provider	{"uid": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "name": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment"}
2024-08-28T04:35:19Z	DEBUG	provider-azure	Async create ended.	{"trackerUID": "66c9728e-c4d8-49f1-83a9-5a283bbafef8", "resourceName": "my-test", "gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "error": "async create failed: failed to create the resource: [{0 A resource with the ID \"/subscriptions/<SUB_ID>/providers/Microsoft.Authorization/roleAssignments/11f7c29f-df3c-4b4f-99cb-a41e2f951bec\" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for \"azurerm_role_assignment\" for more information.  []}]", "tfID": ""}
2024-08-28T04:35:19Z	DEBUG	provider-azure	Reconcile request has been requeued.	{"gvk": "authorization.azure.upbound.io/v1beta1, Kind=RoleAssignment", "name": "my-test", "rateLimiterName": "asyncCallback", "when": "5ms"}

Crossplane beta trace for UserAssignedIdentity:

NAME                                               RESOURCE   SYNCED   READY   STATUS
UserAssignedIdentity/user-assigned-identity-test              False    False   ReconcileError: create failed: async create failed: failed to create the resource: [{0 A resource with the ID "/subscriptions/b54258ce-5c95-43bf-8316-1c07ce0b061d/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test" already exists - to be managed via Terraform this resource needs to be im
ported into the State. Please see the resource documentation for "azurerm_user_assigned_identity" for more information. A resource with the ID "/subscriptions/b54258ce-5c95-43bf-8316-1c07ce0b061d/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource
 documentation for "azurerm_user_assigned_identity" for more information. []}]

Logs from provider-azure-managedidentity after force deleting pod:

2024-08-28T04:30:35Z	DEBUG	provider-azure	Starting	{"sync-interval": "1h0m0s", "poll-interval": "10m0s", "poll-jitter": "30s", "max-reconcile-rate": 100}
2024-08-28T04:30:37Z	INFO	provider-azure	Beta feature enabled	{"flag": "EnableBetaManagementPolicies"}
2024-08-28T04:30:37Z	DEBUG	provider-azure	Calling the inner handler for Create event.	{"gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "name": "user-assigned-identity-test", "queueLength": 0}
2024-08-28T04:30:37Z	DEBUG	provider-azure	Reconciling	{"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}}
2024-08-28T04:30:37Z	DEBUG	provider-azure	Connecting to the service provider	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Instance state not found in cache, reconstructing...	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Observing the external resource	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Diff detected	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"client_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"location\":*terraform.ResourceAttrDiff{Old:\"\", New:\"eastus2\", NewComputed:false, NewRemoved:false, NewExtra:\"eastus2\", RequiresNew:true, Sensitive:false, Type:0x0}, \"name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"user-assigned-identity-test\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"principal_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}, \"resource_group_name\":*terraform.ResourceAttrDiff{Old:\"\", New:\"app-1\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:true, Sensitive:false, Type:0x0}, \"tenant_id\":*terraform.ResourceAttrDiff{Old:\"\", New:\"\", NewComputed:true, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Async create starting...	{"trackerUID": "292dcb81-22bc-4d20-b2de-142ab1083004", "resourceName": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "tfID": ""}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Creating the external resource	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Calling the inner handler for Update event.	{"gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "name": "user-assigned-identity-test", "queueLength": 0}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Successfully requested creation of external resource	{"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}, "uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "version": "571321", "external-name": "", "external-name": ""}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Reconciling	{"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}}
2024-08-28T04:30:42Z	DEBUG	provider-azure	Connecting to the service provider	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:43Z	DEBUG	provider-azure	Instance state not found in cache, reconstructing...	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:43Z	DEBUG	provider-azure	ongoing async operation	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "opType": "create"}
2024-08-28T04:30:43Z	DEBUG	provider-azure	External resource is up to date	{"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}, "uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "version": "571394", "external-name": "", "requeue-after": "2024-08-28T04:40:30Z"}
2024-08-28T04:30:43Z	DEBUG	provider-azure	Reconciling	{"controller": "managed/managedidentity.azure.upbound.io/v1beta1, kind=userassignedidentity", "request": {"name":"user-assigned-identity-test"}}
2024-08-28T04:30:43Z	DEBUG	provider-azure	Connecting to the service provider	{"uid": "292dcb81-22bc-4d20-b2de-142ab1083004", "name": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity"}
2024-08-28T04:30:43Z	DEBUG	provider-azure	Async create ended.	{"trackerUID": "292dcb81-22bc-4d20-b2de-142ab1083004", "resourceName": "user-assigned-identity-test", "gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "error": "async create failed: failed to create the resource: [{0 A resource with the ID \"/subscriptions/<SUB_ID>/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test\" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for \"azurerm_user_assigned_identity\" for more information. A resource with the ID \"/subscriptions/<SUB_ID>/resourceGroups/app-1/providers/Microsoft.ManagedIdentity/userAssignedIdentities/user-assigned-identity-test\" already exists - to be managed via Terraform this resource needs to be imported into the State. Please see the resource documentation for \"azurerm_user_assigned_identity\" for more information. []}]", "tfID": ""}
2024-08-28T04:30:43Z	DEBUG	provider-azure	Reconcile request has been requeued.	{"gvk": "managedidentity.azure.upbound.io/v1beta1, Kind=UserAssignedIdentity", "name": "user-assigned-identity-test", "rateLimiterName": "asyncCallback", "when": "5ms"}

### Crossplane Version

1.14.3

### Provider Version

1.3.0

### Kubernetes Version

1.28.0

### Kubernetes Distribution

AKS, kind

### Additional Info

_No response_

[b-deam]

I forgot to add, if you just use the default management policies this does not occur.

  managementPolicies:
  - "*"

[b-deam]

May also be related; #796