EKS Addon fails on update if resolve_conflicts must be OVERWRITE

[tgillitzer]

What happened?

Updating EKS Addon (coredns). This cluster was created long before we started using Pulumi. Used Pulumi to migrate to the EKS coredns Addon. When this was done, I had to use the OVERWRITE option on resource_conflicts.

Now I am updating the coredns EKS Addon to a new version. I still have the OVERWRITE option set on resource_conflicts.

The update fails, with an error of:
Conflicts found when trying to apply. Will not continue due to resolve conflicts mode.

I turned on debug logging in Pulumi and looked through the log. I see that Pulumi is calling the cloud control API and doing a PATCH. The patch body is here:
{"ClientToken":"<removed>","Identifier":"development|coredns","PatchDocument":"[{\"op\":\"replace\",\"path\":\"/AddonVersion\",\"value\":\"v1.9.3-eksbuild.10\"}]","TypeName":"AWS::EKS::Addon"}

When I do the same operation via the AWS console, it does a POST against /clusters/development/addons/coredns/update and include the ResourceConflicts parameter.

I tried to use the aws CLI and the clouldcontrol command to update the Addon via a PATCH like Pulumi is doing and it doesn't accept ResourceConflicts as a path that is allowed to be update. When I do a cloudcontrol get-resource on it, it doesn't show the ResourceConflicts parameter either.

Maybe this is an AWS Cloud Control problem and not a pulumi problem? Does it make sense for aws-native to use the REST API instead of the CloudFormation API for this?

Thanks

Example

Create a cluster and then add CoreDNS as an addon. My conflict comes from the coredns config map that was created back when the cluster was originally created. If you make a cluster today with the AWS Console, it forces you to deploy the coredns addon and that doesn't result in this same conflict.

        resource_name="coredns-addon",
        addon_name="coredns",
        addon_version="v1.9.3-eksbuild.9",
        cluster_name=cluster.name,
        resolve_conflicts=eks.AddonResolveConflicts.OVERWRITE,
        opts=ResourceOptions(provider=aws_native_provider),
    )

Once the previous update is complete, change the code as below to update the addon to a new version.

        resource_name="coredns-addon",
        addon_name="coredns",
        addon_version="v1.9.3-eksbuild.10",
        cluster_name=cluster.name,
        resolve_conflicts=eks.AddonResolveConflicts.OVERWRITE,
        opts=ResourceOptions(provider=aws_native_provider),
    )

Run update again, and in my case, it fails with the error above.

Output of pulumi about

CLI
Version 3.97.0
Go Version go1.21.5
Go Compiler gc

Plugins
NAME VERSION
aws 6.17.0
aws-native 0.90.0
docker 3.6.1
kubernetes 4.6.1
python unknown

Host
OS Microsoft Windows 11 Pro
Version 10.0.22000 Build 22000
Arch x86_64

This project is written in python: version='3.11.3'

Backend
Name pulumi.com

Dependencies:
NAME VERSION
pip 23.3.1
pulumi-aws 6.17.0
pulumi-aws-native 0.90.0
pulumi-docker 3.6.1
pulumi-kubernetes 4.6.1
setuptools 69.0.2
wheel 0.42.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[mjeffryes]

Thanks for your bug report @tgillitzer. I think this is a subtle interaction between the state management for pulunmi and EKS:

When you change addon_version="v1.9.3-eksbuild.10", pulumi computes the minimal change and sends a PATCH for just the new version. However, based on this documentation I think that EKS probably does not persist the value of resolve_conflicts and needs it to be sent with every update.

This requirement might be implied by ResolveConflicts appearing in the WriteOnlyProperties of the cloud control schema, we'd have to do a bit more research to figure out if this is reasonable behavior for all properties marked "write-only" in the CC schemas.

This bit that you wrote:

I tried to use the aws CLI and the clouldcontrol command to update the Addon via a PATCH like Pulumi is doing and it doesn't accept ResourceConflicts as a path that is allowed to be update. When I do a cloudcontrol get-resource on it, it doesn't show the ResourceConflicts parameter either.

is a bit concerning though. First, I want to check: did you use ResourceConflictsq or ResolveConflicts when you were trying to update via the CLI? (It looks like Cloud Control expects ResolveConflicts) If the cloud control API is rejecting ResolveConflicts on update, that seems like it might reflect a bug in the cloud control API itself which would need to be resolved before we could do anything in the Pulumi provider.

[tgillitzer]

Yeah, based on what I saw, I am pretty sure they expect you to send it with every update. The Admin Console (browser) sends it with every install/update request. It make sense in some was as logically, the "ResolveConflicts" property isn't a property of the addon itself, but of the request to install/update it.

I am pretty sure I used "ResolveConflicts" when I tried to patch via the CLI.

I agree that the issue is at least, if not completely, related to the Cloud Control API. It seems to me they likely didn't factor in exactly how the Addon's work when they designed the Cloud Control API. Or maybe it was generated and thus it doesn't jive with how it actually works. As far as I could tell, there wasn't a way to use the Cloud Control API to update an addon that needed ResolveConflicts set to overwrite.