Expired ECR authorization token causes state file change

[MitchellGerdisch]

What happened?

When using ECR to store the docker image, a temporary token is used and stored in state.
The token expires after 12 hours.
So if a refresh is done after 12 hours, it can't query ECR and wants to remove the image from state.
This is a real problem for customers using Drift Detection schedules with deployments.

Previewing refresh (team-ce/dev)

View in Browser (Ctrl+O): https://app.pulumi.com/team-ce/docker-build-deploy/dev/previews/24558e12-a83a-4bdf-861c-8c477f94a914

     Type                         Name                            Plan       Info
     pulumi:pulumi:Stack          docker-build-deploy-dev                    
     ├─ aws:ecr:Repository        refresh-ecr-repository                     
 -   └─ docker-build:index:Image  refresh_db_export_lambda_image  delete     1 warning

Diagnostics:
  docker-build:index:Image (refresh_db_export_lambda_image):
    warning: failed to get manifest 052848974346.dkr.ecr.us-east-2.amazonaws.com/refresh-ecr-repository-62805af:latest@sha256:81cff66bebd5d44c86e967b25ed0075538165a0041dac4735f9a4a00b1f3b4bd: request failed: unauthorized [http 403]: {"errors":[{"code":"DENIED","message":"Your authorization token has expired. Reauthenticate and try again."}]}

Resources:
    - 1 to delete
    2 unchanged

Do you want to perform this refresh?

Example

• Run the code below.
• Run pulumi refresh and see that it's fine.
• Wait more than 12 hours.
• Run pulumi refresh and see the error above and it's desire to remove the image from state because it can't communicate with ECR.

import pulumi
import pulumi_docker_build as docker_build
import pulumi_aws as aws

base_name="refresh"

ecr_repository = aws.ecr.Repository(
    f"{base_name}-ecr-repository",
    force_delete=True,
)

auth_token = aws.ecr.get_authorization_token_output(registry_id=ecr_repository.registry_id)

ecr_image = docker_build.Image(
            f"{base_name}_db_export_lambda_image",
            context={
                "location": "./app",
            },
            push=True,
            platforms=[
                docker_build.Platform.LINUX_ARM64,
            ],
            tags=[ecr_repository.repository_url.apply(lambda repository_url: f"{repository_url}:latest")],
            build_on_preview=True,
            registries=[
                docker_build.RegistryArgs(
                    address=ecr_repository.repository_url,
                    username=auth_token.user_name,
                    password=auth_token.password,
                )
            ],
        )

pulumi.export("ecr_registry_id", ecr_repository.registry_id)

Output of pulumi about

CLI
Version 3.147.0
Go Version go1.23.5
Go Compiler gc

Plugins
KIND NAME VERSION
resource aws 6.66.3
resource command 1.0.1
resource docker-build 0.0.9
language python 3.147.0

Host
OS darwin
Version 15.2
Arch x86_64

This project is written in python: executable='/Users/mitch/Downloads/wmg-dockerbuild-token-expiry/dockerbuild/venv/bin/python' version='3.11.8'

Current Stack: xxxxxx/docker-build-deploy/dev

TYPE URN
pulumi:pulumi:Stack urn:pulumi:dev::docker-build-deploy::pulumi:pulumi:Stack::docker-build-deploy-dev
pulumi:providers:aws urn:pulumi:dev::docker-build-deploy::pulumi:providers:aws::default_6_66_3
aws:ecr/repository:Repository urn:pulumi:dev::docker-build-deploy::aws:ecr/repository:Repository::refresh-ecr-repository
pulumi:providers:docker-build urn:pulumi:dev::docker-build-deploy::pulumi:providers:docker-build::default_0_0_9
docker-build:index:Image urn:pulumi:dev::docker-build-deploy::docker-build:index:Image::refresh_db_export_lambda_image

Found no pending operations associated with team-ce/dev

Backend
Name pulumi.com

Dependencies:
NAME VERSION
pulumi_aws 6.66.3
pulumi_command 1.0.1
pulumi_docker_build 0.0.9
setuptools 65.5.0

Additional context

No response

Contributing

Vote on this issue by adding a 👍 reaction.
To contribute a fix for this issue, leave a comment (and link to your pull request, if you've opened one already).

[rquitales]

Thanks for reporting this. This appears to be an unfortunate side-effect of the current design of this resource. While an ideal solution would require further enhancements to our core engine, we might be able to relax deletion in our diff logic. I'll investigate further to see if this is a viable workaround.

[EronWright]

See also: pulumi/pulumi-awsx#1537

[mjeffryes]

As per the discussion in the issue @EronWright linked, this essentially boils down to another enhancement request for pulumi/pulumi#16600 (We have to run the program to get fresh credentials from ECR.)

[EronWright]

Per latest conversation, I expect we need a different solution to this problem. I'll write a proposal about some possible design changes to this provider.