-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
:instance_exists waiter not waiting for existence #859
Comments
The |
Then this seems like an issue with the Amazon EC2 API. It is returning a message saying that 'yes, the instance does exist' but then fails when trying to update that existence. Perhaps there are multiple meanings of the word Whatever the answer is, the current state of this API is still very confusing. |
As a follow up to this, also found this code block failing: server = ec2.create_instance(instance_data)
ec2.client.wait_until(
:instance_exists,
:instance_ids => [server.id]
) The But isn't that why I'm calling the waiter? I want it to poll until that instance exists? |
The ec2.wait_until(:instance_exists, instance_ids:['i-12345678'])
[Aws::EC2::Client 400 0.794136 0 retries] describe_instances(instance_ids:["i-12345678"]) Aws::EC2::Errors::InvalidInstanceIDNotFound The instance ID 'i-12345678' does not exist
[Aws::EC2::Client 400 0.459063 0 retries] describe_instances(instance_ids:["i-12345678"]) Aws::EC2::Errors::InvalidInstanceIDNotFound The instance ID 'i-12345678' does not exist
[Aws::EC2::Client 400 0.370968 0 retries] describe_instances(instance_ids:["i-12345678"]) Aws::EC2::Errors::InvalidInstanceIDNotFound The instance ID 'i-12345678' does not exist This makes me wonder if Amazon EC2 is sometimes returning a different error code. Can you rescue the Aws::Waiters::Errors::UnexpectedError, and then log the original error: begin
# call waiter here
rescue Aws::Waiters::Errors::UnexpectedError => unexpected
puts unexpected.error.class.name
puts unexpected.error.message
end |
We are using version 2.1.11 (just for more information). I'll add the block you suggested and see if I can get some more information, but you can see test-kitchen/kitchen-ec2#184 and grep the page for After re-reading our two stack traces it looks like we have seen two different exceptions. First we saw Then I switched from using Now we see the error |
The Aws::EC2::Errors::InvalidInstanceIDNotFoundThis error should automatically trigger another polling attempt. If the configured number of polling attempts lapses, then an Aws::Waiters::Errors::UnexpectedErrorThis error is reserved for service responses that contain an error which is not defined in the waiter definition. An error can trigger a waiter success, fail, or retry state. Any unexpected errors are wrapped in an instance of How it should workBased on the expected behavior, your should not be getter either of these errors, and it should continue polling. Additionally, I am unable to reproduce the issue myself. See the following examples: This example makes up a fake instance ID and attempts to poll. This one polls the configured number of attempts and then gives up: ec2 = Aws::EC2::Resource.new
instance = ec2.instance('i-12345678')
instance.wait_until_exists do |waiter|
waiter.max_attempts = 2
waiter.delay = 1
end
# raises Aws::Waiters::Errors::TooManyAttemptsError after 2 failed attempts This one starts a new instance and polls it straightway: ec2 = Aws::EC2::Resource.new
instances = ec2.create_instances({
image_id: 'ami-1ccae774',
instance_type: 'm1.large',
min_count: 1,
max_count: 1
})
instance = instances.first
instance.wait_until_exists
instance.terminate |
I figured out why I was seeing But I still see my original issue - the I understand that the instance may not be taggable yet, but it still seems very weird that it is returning a |
The `:instance_running` and `:instance_state_ok` waiters were raising an unexpeced error, i.e. `Aws::EC2::Errors::InvalidInstanceIDNotFound` instead of retrying. This could happen only newly launched instances. See #859.
Good catch. I've added logic to the As for the originally reported issue, it seems unfortunate that there is not an obvious way, via the Amazon EC2 API to know when an instance may be tagged. I'm going to play around with the DescribeTags API operation and see if that might provide more information. |
@trevorrowe Thanks so much for all the support! |
At this point, I'm inclined to close this issue. I will certainly watch out for the opportunity to improve this, but currently there appears to be no deterministic way to wait until the instance is tagable without simply trying, rescuing the error and retrying until the system is internally consistent. |
I'm confused by some behavior I just saw. I'm using the SDK V2. In test kitchen I send a
create_instance
request and immediately execute a wait_until:instance_exists
. After that I try to tag the instance.I just got the following error during this -
Aws::EC2::Errors::InvalidInstanceIDNotFound
The instance ID 'i-b3dbea44' does not exist
.Am I using the wrong waiter? I would not expect to get a
does not exist
error immediately after using theinstance_exists
waiter.The text was updated successfully, but these errors were encountered: