aws cloudformation deploy command could not wait stack create/update more than 10 minutes

[mvinii94]

Hi,

I am doing some tests by my side and I've run the aws cli in debug mode trying to find why the aws cli commands hangs up on long CloudFormation create/update processes, I could find that the 'aws cloudformation deploy' command checks the CloudFormation Stack status every 5 seconds and then, after approximately 10 minutes and 30 seconds the code returns the error:

--- Error ---
"WaiterError: Waiter StackCreateComplete failed: Max attempts exceeded"
--- Error ---

I think the check period could be greater, then the aws cli command could wait more time, or increase the number of max attempts, but I think the first option it's better. Could you please take a look at this?

tests:

time aws cloudformation deploy --template-file cfn-rolling-deployment.yaml --stack-name rollindeployment-test --capabilities CAPABILITY_NAMED_IAM
Waiting for changeset to be created..
Waiting for stack create/update to complete

Failed to create/update the stack. Run the following command
to fetch the list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name rollindeployment-test
aws cloudformation deploy --template-file cfn-rolling-deployment.yaml 0.90s user 0.13s system 0% cpu 10:29.37 total

---

Traceback:

2017-08-09 10:48:41,571 - MainThread - botocore.hooks - DEBUG - Event needs-retry.cloudformation.DescribeStacks: calling handler <botocore.retryhandler.RetryHandler object at 0x10e3ac550>
2017-08-09 10:48:41,571 - MainThread - botocore.retryhandler - DEBUG - No retry needed.
2017-08-09 10:48:41,571 - MainThread - awscli.customizations.cloudformation.deployer - DEBUG - Execute changeset waiter exception
Traceback (most recent call last):
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 177, in wait_for_execute
waiter.wait(StackName=stack_name)
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/botocore/waiter.py", line 53, in wait
Waiter.wait(self, **kwargs)
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/botocore/waiter.py", line 327, in wait
last_response=response
WaiterError: Waiter StackCreateComplete failed: Max attempts exceeded
2017-08-09 10:48:41,573 - MainThread - awscli.clidriver - DEBUG - Exception caught in main()
Traceback (most recent call last):
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/clidriver.py", line 200, in main
return command_table[parsed_args.command](remaining, parsed_args)
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/clidriver.py", line 338, in call
return command_table[parsed_args.operation](remaining, parsed_globals)
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/commands.py", line 187, in call
return self._run_main(parsed_args, parsed_globals)
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 160, in _run_main
parsed_args.execute_changeset)
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deploy.py", line 172, in deploy
deployer.wait_for_execute(stack_name, result.changeset_type)
File "/usr/local/Cellar/awscli/1.11.129/libexec/lib/python2.7/site-packages/awscli/customizations/cloudformation/deployer.py", line 181, in wait_for_execute
raise exceptions.DeployFailedError(stack_name=stack_name)
DeployFailedError: Failed to create/update the stack. Run the following command
to fetch the list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name rollindeployment-test2
2017-08-09 10:48:41,575 - MainThread - awscli.clidriver - DEBUG - Exiting with rc 255

Failed to create/update the stack. Run the following command
to fetch the list of events leading up to the failure
aws cloudformation describe-stack-events --stack-name rollindeployment-test2
aws cloudformation deploy --template-file cfn-rolling-deployment.yaml 0 1.21s user 0.16s system 0% cpu 10:31.42 total

[dimaulupov]

It happens on all versions past 1.11.117.

Any cloudformation deploy fails at 10 min mark.

[joguSD]

Yes, this bug was introduced in this PR #2686. Which lines up with the past 1.11.117 claim. More discussion and a potential fix is over there.

[Seblat5ch]

The issue is because the poller happens every 5 seconds waiting for a CREATE_COMPLETE

In the below pull request you will find all the information
#2761