-
Notifications
You must be signed in to change notification settings - Fork 4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
‼️ NOTICE: last modified date does not match with the last modified date of the retrieved parameters #14629
Comments
I have had this problem twice and after upgrading to v1.103 it seems to be solved. Thank you very much. |
We have this issue after upgrading to v1.103, but haven't encountered it before. We like to keep our dependencies up to date, so we have updated to every minor release in recent history. The change set that failed has this parameter:
That parameter is referenced as the |
We had this happen on another stack that has not been updated to v1.103 yet. It's still using v1.102. I don't know if this helps, but in this stack the |
I have this issue, and I'm on 1.103
I believe it has to do with the AMI
|
@nirradi two quick things to validate:
|
I see now that the two systems which I work on differ by the version. Its working on 1.101 (both python packages and cli) I'm not inclined to share the entire cfn, here are the diffs between 1.101 and 1.103
|
@nirradi if this is failing, With v1.103.0, can you give us the output of |
Mentioned this in the other issue but just a friendly reminder so it doesn't fall through the cracks: I'm happy to try this on the 2.0 line (I recently ported to from 1.x to 2 RC1) but I'm waiting for the 2.0 release to come through (last one was ~12 days ago). |
I resolved this issue on my EKS stacks by manually updating a stack as referenced here: #14467 (comment) with a stack template generated with 1.103.0. Subsequent |
@rsyabuta to clarify, cdk deploy on v1.103.0 was still failing with the param issue and required deploying without a change set? |
Yes that is correct. Here is the
And the
|
|
As I've mentioned in #14467 (comment), I encounter this in the The reference in the stack is simple: this.bastion = new ec2.BastionHostLinux(this, `Bastion`, {
vpc: props.vpc,
instanceName: `${props.name}-bastion`,
securityGroup: bastionSg,
subnetSelection: {
subnetType: ec2.SubnetType.PUBLIC,
},
}); Diff for the stack: cdk diff vpc
Stack vpc (vpc)
Parameters
[+] Parameter SsmParameterValue:--aws--service--ami-amazon-linux-latest--amzn2-ami-hvm-x86_64-gp2:C96584B6-F00A-464E-AD19-53AFF4B05118.Parameter SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter: {"Type":"AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>","Default":"/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2"}
Resources
[~] AWS::EC2::Instance Bastion/Bastion/Resource BastionBastion34649AD2 replace
└─ [~] ImageId (requires replacement)
└─ @@ -1,1 +1,3 @@
[-] "{{resolve:ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2}}"
[+] {
[+] "Ref": "SsmParameterValueawsserviceamiamazonlinuxlatestamzn2amihvmx8664gp2C96584B6F00A464EAD1953AFF4B05118Parameter"
[+] }
[~] AWS::EC2::EIP Bastion/BastionEIP BastionBastionEIP54D072EC may be replaced
└─ [~] InstanceId (may cause replacement)
└─ [~] .Ref:
├─ [-] BastionBastion34649AD2
└─ [+] BastionBastion34649AD2 (replaced) Fails with: vpc (vpc): deploying...
vpc: creating CloudFormation changeset...
❌ vpc (vpc) failed: Error: Failed to create ChangeSet cdk-deploy-change-set on -vpc: FAILED, Parameters: [ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2:45:1619654085749] last modified date does not match with the last modified date of the retrieved parameters.
at /home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:227:11
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at waitFor (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:188:20)
at Object.waitForChangeSet (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/util/cloudformation.ts:213:15)
at Object.deployStack (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/api/deploy-stack.ts:260:32)
at CdkToolkit.deploy (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/lib/cdk-toolkit.ts:180:24)
at initCommandLine (/home/malanius/.nvm/versions/node/v16.0.0/lib/node_modules/aws-cdk/bin/cdk.ts:209:9)
Failed to create ChangeSet cdk-deploy-change-set on -vpc: FAILED, Parameters: [ssm:/aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2:45:1619654085749] last modified date does not match with the last modified date of the retrieved parameters. The workaround with manually updating stack from generated template by 1.103.0 worked for me. Probably just temporarily deleting the offending bastion resource could work as well, but this way I at least avoided updating all the stacks where Bastion is mentioned. |
I had this issue repeatedly on 1.101 and 1.102 and my solution was to delete and redeploy the stack when it happened. The problem went away on it's own before this issue was opened without me having to upgrade. I believe the last time I hit it was May 6. I have deployed at least 20 updates in the last few days on that 1.102 stack with no issues. |
Does the new I have Error:
I think the parameter in question is this one:
However, I'm not explicitly referencing this myself AFAIK. Just double checked and I can't see anywhere where I am referencing anything like this. I tried updating the stack via the web console, using the existing template without making any changes at all (is that the right process?) but I got the error:
|
Also ran into this with the |
Updated the issue to include manually deploying your stack without a changeset as a workaround while we figure out if/how to allow users to do this via CDK. For users who have CI/CD deploying a high volume of stacks this obviously isn't ideal and we are figuring out if we want to add a way for the CDK CLI to deploy without a changeset and if that is good enough. If people are using the codepipeline stack deploy actions with a changeset this won't cover them and I'm not sure if there is another way to let them easily workaround this. |
In my case I just upgraded to The deployment finished fine. However, the very next deployment failed.
So it seems that actually upgrading to
The "faulty" parameter seems to be added automatically as part of creating an ECS cluster. It is not something I'm adding. |
We are running into this after upgrading to 1.103.0 today, seemingly caused by usage of Error is Updating the stack with a 1.103.0 generated template fixed the issue for us. |
@andreialecu I don't see how that is possible. What version were you on when the stack was last deployed? Can you also verify that you were using both the CLI and @aws-cdk/* libraries with version 1.103.0? It appears that not only dynamic references were broken, but all SSM param references including those passed via CFN parameters. If that is the case, the timing of the breakage could be based purely on when these parameters were updated in relation to when stack updates occurred and when the feature was rolled out on CFN's side. This makes it really hard for us to figure out what users are affected and how exactly. Are you able to attempt updating your stack manually as mentioned in the issue to workaround? I'm curious if that helps in your case. |
@MrArnoldPalmer It was only after updating to 103 that the diff appeared, as mentioned in my previous comment. Deploying it the first time worked. But any subsequent deploys have failed. I haven't yet tried manually deploying. |
Oops. My screenshot does show that it was actually previously deployed on the 5th of May, probably using 1.100.0. NOT in March. Sorry about that. Had a mental hiccup, I thought 05.05 was today 🤦🏻♂️ |
@MrArnoldPalmer I can confirm that manually updating the stack using the AWS console did work. |
Thank you @andreialecu for investigating! Anyone else that is still blocked and having trouble with the workarounds please keep letting us know. |
Thanks for the response @MrArnoldPalmer 😃 I am using the CI/CD AWS CDK Pipeline to deploy my stacks. Just confirming what you're saying here, there is not currently a workaround? Would rolling back the cdk version work? |
I managed to go long time without this issue but it came back today. I upgraded from 1.101.0 to 1.104.0 but it did not fix it. This is for an ECS cluster and the parameter is implicit in the construct as opposed to anything I create in my own code: I used the workaround of cdk synth and manually uploaded the template via the console and that is working. |
Upgrading from 1.102.0 to 1.104.0 did not solve my issue but the workaround provided by @cseickel worked. |
@seawatts unfortunately no, this isn't casued by anything in the CDK it's caused by an issue on the cloudformation side. The only workaround is to run |
Is the upgrade to cdk 1.103 necessary for the workaround? Would simply running |
Unfortunately, none of the suggestions in this and other linked tickets have worked for me. I've tried:
Code: Are there any other suggestions to try other than deleting/recreating stacks? |
|
@clemans Thanks! Your reply made me realize I hadn't tried uploading via the AWS Management Console, only command line. The upload via the console worked! Saves me from deleting / recreating stacks. Appreciate the help! |
Glad the console resolved this though that seems like the easiest if you only need to do a couple of stacks. |
@rahulpshah updating is not required to do the workaround, but it may prevent you from needing it as some users report v1.103.0 fixing the issue for them. |
Also getting this issue for |
I am using AWS SAM template. I believe, underneath, it is using aws cdk. I am getting same issue. I don't have an option of deletion of the stack or doing it manual. Do we have any workaround for aws sam CLI? Thanks |
I managed to find a fix for this with @Shogan . It is a "surgical" workaround , but will allow you to get the stack in an "updateable" state using CDK without recreating it.
If all goes well , you should be able to deploy the stack using cdk going forward. |
I did something similar to @AbdoNile but simply replaced the parameters with the actual values. After that succeeded, I deployed again through CDK and it worked. |
This issue was a notification for a transient error. Due to the lack of recent engagement, I'll assume most people that were affected by this managed to unstick themselves by now. Closing. |
|
Please add your +1 👍 to let us know you have encountered this
Status: INVESTIGATING
Overview:
Cloudformation recently deployed a change to allow dynamic references to ssm parameters within templates without needing to provide a version. This allows always referencing the latest value of a parameter within your template. This change had a bug that caused change set creation to fail when the parameter value was updated.
This breakage was caused by the references resolving to a version of the parameter that include a timestamp. Upon changeset creation, if the parameter has been updated, cloudformation sees the parameter name and the previously resolved timestamped version as different values and fails to update. They have since changed logic so that the strings resolved do not include timestamps, however if a template was successfully deployed during the time that the timestamp was being appended, a stack cannot be updated via changeset while referencing a dynamic parameter.
UPDATE:
It is clear that removing all dynamic ssm references doesn't cause users to get unstuck. Users have to manually deploy their stacks without a changeset to get past this. That being the case, for users with lots of stacks updating each manually is a pain. We are considering adding an option to the CLI to skip a changeset if this error occurs during deploy.
Complete Error Message:
Workaround:
Attempt the following steps in order as a workaround:
Upgrade both your CLI and all your @aws-cdk/xyz modules to v1.103.0 - This removes dynamic SSM references and goes back to referencing them via CFN parameters.
Delete your stack and redeploy - If you are unable to do so because this is a production stack, and upgrading to v1.103 didn't fix the issue, please comment here with the following information and try number 3.
Run
cdk synth
and then manually deploy this stack using the the aws cli or cfn console - This is not an ideal workaround for users who may have a lot of different stacks that need updating but in the meantime, this can unblock you if you need a stack deployed. We are working on figuring out another option that will allow skipping changeset creation via the aws cli.Solution:
This issue only shows up for users who deployed stacks during a short period of time before cloudformation fixed some of these bugs. Because of that, it is hard for us to test ourselves if removal of dynamic param references (#14527) solves this. If it doesn't, another solution mentioned here is to update your stack without creating a changeset. We may be able to provide an option in the CLI to skip a changeset if this error is encountered. Since we don't have broken stacks to test this with, please let us know if this is or an additional fix is needed.
Related Issues:
#14467
aws/aws-cli#6106
The text was updated successfully, but these errors were encountered: