Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Cloud-init fails on AWS if IMDSv2 returns a 503 error. #5577

Open
taylorpn opened this issue Aug 1, 2024 · 1 comment
Open

Cloud-init fails on AWS if IMDSv2 returns a 503 error. #5577

taylorpn opened this issue Aug 1, 2024 · 1 comment
Labels
bug Something isn't working correctly ds: ec2 Issues specific to DataSourceEc2

Comments

@taylorpn
Copy link

taylorpn commented Aug 1, 2024

Bug report

Cloud-init does not retry IMDSv2 token calls when 503 is returned. According to the public documentation, a 503 error code should be retried and not result in a failure in and of itself.
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html

Steps to reproduce the problem

The problem cannot be reproduced on demand.

Environment details

  • Cloud-init version: 19.3 (All versions suspected.)
  • Operating System Distribution: Amazon Linux 2023 (But could happen on any cloud-init-enabled OS running on AWS.)
  • Cloud provider, platform or installer type: Amazon Web Services

cloud-init logs

2024-07-24 08:13:06,797 - DataSourceEc2.py[DEBUG]: Fetching Ec2 IMDSv2 API Token
2024-07-24 08:13:06,797 - url_helper.py[DEBUG]: [0/1] open 'http://169.254.169.254/latest/api/token' with {'url': 'http://169.254.169.254/latest/api/token', 'allow_redirects': True, 'method': 'PUT', 'timeout': 50.0, 'headers': {'User-Agent': 'Cloud-Init/22.2-0ubuntu1~20.04.3', 'X-aws-ec2-metadata-token-ttl-seconds': 'REDACTED'}} configuration
2024-07-24 08:13:06,801 - url_helper.py[DEBUG]: Read from http://169.254.169.254/latest/api/token (503, 359b) after 1 attempts
2024-07-24 08:13:06,801 - url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/120s]: bad status code [503]
2024-07-24 08:13:06,801 - DataSourceEc2.py[WARNING]: Fatal error while requesting Ec2 IMDSv2 API tokens
2024-07-24 08:13:06,801 - DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled

[   15.417209] cloud-init[10775]: Feb 12 11:15:27 cloud-init[10775]: url_helper.py[WARNING]: Calling 'http://169.254.169.254/latest/api/token' failed [0/120s]: bad status code [503]
[   15.417267] cloud-init[10775]: Feb 12 11:15:27 cloud-init[10775]: DataSourceEc2.py[WARNING]: Fatal error while requesting Ec2 IMDSv2 API tokens
[   15.417864] cloud-init[10775]: Feb 12 11:15:27 cloud-init[10775]: DataSourceEc2.py[WARNING]: IMDS's HTTP endpoint is probably disabled
[   15.467706] cloud-init[10775]: Feb 12 11:15:27 cloud-init[10775]: cc_write_metadata.py[WARNING]: there is no identity dataset
[   15.468057] cloud-init[10775]: Feb 12 11:15:27 cloud-init[10775]: util.py[WARNING]: Running module write-metadata (<module 'cloudinit.config.cc_write_metadata' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_write_metadata.pyc'>) failed
[[1;31mFAILED[0m] Failed to start Initial cloud-init job (metadata service crawler).
@taylorpn taylorpn added bug Something isn't working correctly new An issue that still needs triage labels Aug 1, 2024
@holmanb
Copy link
Member

holmanb commented Aug 2, 2024

Thanks for reporting, @taylorpn!

According to the public documentation, a 503 error code should be retried and not result in a failure in and of itself.

Yeah, I think that this would generally be safe to do.

holmanb added a commit to holmanb/cloud-init that referenced this issue Aug 2, 2024
@holmanb
feat(url_helper): Retry on 503 error
cc16534 
If the server is busy, no need to fail.
Add type hints to adjascent code paths.

Fixes canonicalGH-5577
@holmanb holmanb mentioned this issue Aug 2, 2024
Closed
2 tasks
@aciba90 aciba90 removed the new An issue that still needs triage label Aug 2, 2024
@github-actions github-actions bot added the Stale label Sep 6, 2024
@aciba90 aciba90 added ds: ec2 Issues specific to DataSourceEc2 and removed Stale labels Oct 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working correctly ds: ec2 Issues specific to DataSourceEc2
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat(url_helper): Retry on 503 error holmanb/cloud-init
3 participants
@holmanb @aciba90 @taylorpn