TeamMembership etag causes endless reconciliation loop and rate limit hit

[irizzant]

@AtzeDeVries
I've created a TeamMembership resource in Crossplane and as far as I can see the provider logs endless reconciliation logs like these:

2024-12-23T10:47:56Z    DEBUG   provider-upjet-sgithub  Successfully requested update of external resource      {"controller": "managed/team.github.upbound.io/v1alpha1, kind=teammembership", "request": {"name":"example-project-project-irizzant"}, "uid": "108a1887-ce65-4f2c-8796-6cc20c461820", "version": "28129", "external-name": "11812739:irizzant", "requeue-after": "2024-12-23T10:58:23Z"}
2024-12-23T10:47:56Z    DEBUG   provider-upjet-sgithub  Async update starting...        {"trackerUID": "108a1887-ce65-4f2c-8796-6cc20c461820", "resourceName": "example-project-project-irizzant", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership", "tfID": "11812739:irizzant"}
2024-12-23T10:47:56Z    DEBUG   provider-upjet-sgithub  Updating the external resource  {"uid": "108a1887-ce65-4f2c-8796-6cc20c461820", "name": "example-project-project-irizzant", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership"}
2024-12-23T10:47:57Z    DEBUG   provider-upjet-sgithub  Reconciling     {"controller": "managed/repo.github.upbound.io/v1alpha1, kind=repositoryfile", "request": {"name":"example-project-apps-example-envs-prod-eu-central-1-replicas"}}
2024-12-23T10:47:57Z    DEBUG   provider-upjet-sgithub  Connecting to the service provider      {"uid": "b5539471-5082-4bd3-866c-765320eb3ed0", "name": "example-project-apps-example-envs-prod-eu-central-1-replicas", "gvk": "repo.github.upbound.io/v1alpha1, Kind=RepositoryFile"}
2024-12-23T10:47:57Z    DEBUG   provider-upjet-sgithub  Async update ended.     {"trackerUID": "b36cee4f-b357-476f-843e-dcf081db975b", "resourceName": "example-project-project-clementebiondo", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership", "error": null, "tfID": "11812739:clementebiondo"}
2024-12-23T10:47:57Z    DEBUG   provider-upjet-sgithub  Observing the external resource {"uid": "b5539471-5082-4bd3-866c-765320eb3ed0", "name": "example-project-apps-example-envs-prod-eu-central-1-replicas", "gvk": "repo.github.upbound.io/v1alpha1, Kind=RepositoryFile"}
2024-12-23T10:47:57Z    DEBUG   provider-upjet-sgithub  Reconcile request has been requeued.    {"gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership", "name": "example-project-project-clementebiondo", "rateLimiterName": "", "when": "0s"}
2024-12-23T10:47:57Z    DEBUG   provider-upjet-sgithub  Reconciling     {"controller": "managed/team.github.upbound.io/v1alpha1, kind=teammembership", "request": {"name":"example-project-project-clementebiondo"}}
2024-12-23T10:47:57Z    DEBUG   provider-upjet-sgithub  Connecting to the service provider      {"uid": "b36cee4f-b357-476f-843e-dcf081db975b", "name": "example-project-project-clementebiondo", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership"}
2024-12-23T10:47:58Z    DEBUG   provider-upjet-sgithub  Observing the external resource {"uid": "b36cee4f-b357-476f-843e-dcf081db975b", "name": "example-project-project-clementebiondo", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership"}
2024-12-23T10:47:58Z    DEBUG   provider-upjet-sgithub  Async update ended.     {"trackerUID": "108a1887-ce65-4f2c-8796-6cc20c461820", "resourceName": "example-project-project-irizzant", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership", "error": null, "tfID": "11812739:irizzant"}
2024-12-23T10:47:58Z    DEBUG   provider-upjet-sgithub  Reconcile request has been requeued.    {"gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership", "name": "example-project-project-irizzant", "rateLimiterName": "", "when": "0s"}
2024-12-23T10:47:58Z    DEBUG   provider-upjet-sgithub  Reconciling     {"controller": "managed/team.github.upbound.io/v1alpha1, kind=teammembership", "request": {"name":"example-project-project-irizzant"}}
2024-12-23T10:47:58Z    DEBUG   provider-upjet-sgithub  Connecting to the service provider      {"uid": "108a1887-ce65-4f2c-8796-6cc20c461820", "name": "example-project-project-irizzant", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership"}
2024-12-23T10:47:58Z    DEBUG   provider-upjet-sgithub  Diff detected   {"uid": "b36cee4f-b357-476f-843e-dcf081db975b", "name": "example-project-project-clementebiondo", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"role\":*terraform.ResourceAttrDiff{Old:\"maintainer\", New:\"member\", NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}

This ends up with hitting GitHub API rate limit and prevents the provider from working.

After digging further into it, I found that the resourceVersion was continuously changing too, and with that the status.atProvider.etag field.

There is an open issue for that in integrations/terraform-provider-github#796 , but there is no workaround because the etag field is not part of the terraform resource field and so there is no way to tell Crossplane to ignore this with Crossplane initProvider.

I also tried with managementPolicies: ["Observe", "Create", "Update", "Delete"] but the provider still goes through endless reconciliation.

[irizzant]

Closing my own issue because the answer was right in the log file:

2024-12-23T10:47:58Z    DEBUG   provider-upjet-sgithub  Diff detected   {"uid": "b36cee4f-b357-476f-843e-dcf081db975b", "name": "example-project-project-clementebiondo", "gvk": "team.github.upbound.io/v1alpha1, Kind=TeamMembership", "instanceDiff": "*terraform.InstanceDiff{mu:sync.Mutex{state:0, sema:0x0}, Attributes:map[string]*terraform.ResourceAttrDiff{\"role\":*terraform.ResourceAttrDiff{**Old:\"maintainer\", New:\"member\",** NewComputed:false, NewRemoved:false, NewExtra:interface {}(nil), RequiresNew:false, Sensitive:false, Type:0x0}}, Destroy:false, DestroyDeposed:false, DestroyTainted:false, RawConfig:cty.NilVal, RawState:cty.NilVal, RawPlan:cty.NilVal, Meta:map[string]interface {}(nil)}"}

and in particular:

Old:\"maintainer\", New:\"member\",

The user is a organization owner and if you try to assign it a different permission GitHub will set it back to maintainer thus triggering an infinite loop

[AtzeDeVries]

@irizzant Thanks! Great that you found the solution yourself.