Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Setting EC2 instance http_endpoint/token metadata_options breaks teleport config #10962

Closed
deusxanima opened this issue Mar 8, 2022 · 6 comments
Closed

Setting EC2 instance http_endpoint/token metadata_options breaks teleport config #10962

deusxanima opened this issue Mar 8, 2022 · 6 comments
Assignees
@fheinecke
Labels
bug c-vw Internal Customer Reference

Comments

@deusxanima
Copy link
Contributor

deusxanima commented Mar 8, 2022 •
edited by programmerq
Loading

Description

What happened:
After setting this on the proxy launch configuration:

metadata_options {
   http_endpoint = "enabled"
   http_tokens   = "required"
}

Customer's teleport configuration gets injected with raw html and mangles the customer config as seen below:

teleport:
  auth_token: /var/lib/teleport/token
  ca_pin: << REDACTED >>
  nodename: <?xml version="1-0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1-0 Transitional//EN"
        "http://www-w3-org/TR/xhtml1/DTD/xhtml1-transitional-dtd">
<html xmlns="http://www-w3-org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
 </head>
 <body>
  <h1>401 - Unauthorized</h1>
 </body>
</html>
  advertise_ip: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
 </head>
 <body>
  <h1>401 - Unauthorized</h1>
 </body>
</html>
  log:
    output: stderr
    severity: INFO
  data_dir: /var/lib/teleport

It appears that the issue is related to the way the we're setting/using the IMDS_TOKEN_HEADER

Customer testing showed that switching from the following:

IMDS_TOKEN=$(curl -m5 -sS -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
IMDS_TOKEN_HEADER="-H \"X-aws-ec2-metadata-token: ${IMDS_TOKEN}\""
curl -m5 -sS "${IMDS_TOKEN_HEADER}" ${CURL_EXTRA_ARGS} [http://169.254.169.254/latest/${REQUEST_PATH}](http://169.254.169.254/latest/$%7BREQUEST_PATH%7D)

to the following:

IMDS_TOKEN=$(curl -m5 -sS -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 300")
curl -m5 -sS -H "X-aws-ec2-metadata-token: ${IMDS_TOKEN}" ${CURL_EXTRA_ARGS} [http://169.254.169.254/latest/${REQUEST_PATH}](http://169.254.169.254/latest/$%7BREQUEST_PATH%7D)

seems to have resolved the problem in the customer environment.

What you expected to happen:
Should be able to set instance metadata as specified above and not corrupt the teleport.yaml config

Server Details

  • Teleport version (run teleport version):

gz#4227
@deusxanima deusxanima added bug c-vw Internal Customer Reference labels Mar 8, 2022
@fheinecke fheinecke mentioned this issue Jun 13, 2022
Merged
@fheinecke
Copy link
Contributor

Confirmed and PR opened (#13417). Thanks for the great bug report!

@fheinecke
Copy link
Contributor

Close as fixed via PRs #13417, #13538, #13537, and #13704.

@r0mant
Copy link
Collaborator

r0mant commented Jul 12, 2022

@AHARIC I'm going to reopen for now since you mentioned customer is still experiencing this issue, however please verify with them first that they're using the new version. The PR basically implements the change that the ticket description says fixes the issue for them. It should be available in 9.3.9.

cc @fheinecke FYI

@r0mant r0mant reopened this Jul 12, 2022
@webvictim
Copy link
Contributor

Is this the same issue as #14359?

@fheinecke
Copy link
Contributor

@webvictim Strictly speaking I don't believe so. They appear interrelated but the only way this code is hit is if the assets/aws/files/bin/teleport-generate-config script is called. Even if the client in was to be using this inadvertently somewhere there are a number of other places that the script would likely break first.

I am reasonably certain that the root cause of #14359 is https://github.com/gravitational/teleport/blob/master/lib/utils/ec2.go#L128. It looks like when Teleport starts up it calls that function to determine if the host is an EC2 instance, then calls sets the node name if true: https://github.com/gravitational/teleport/blob/master/lib/service/service.go#L853-L854

The check in ec2.go only verifies that Teleport can make a HTTP request to http://169.254.169.254/latest/meta-data with a 200 response. There are several problems with this, namely that AWS is not the only provider to have metadata at this address/path. Here are some other providers that provide the data at that endpoint and/or path:

  • Azure
  • DigitalOcean
  • E24Cloud
  • Exoscale
  • GCP (to a much lessor extent)
  • MAAS
  • OpenStack
  • Oracle (not 100% sure on this one)
  • UpCloud
  • Vultr
  • ZStack

Additionally the GetTagValue function in ec2.go needs a lot more error checking to verify that what is read is actually what is expected to prevent similar bugs in the future.

@russjones
Copy link
Contributor

Since we can't reproduce this anymore and it's been fixed, I am going to close this ticket. Please create another ticket if this still occurs in Teleport 10.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@fheinecke fheinecke

Labels
bug c-vw Internal Customer Reference
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@r0mant @webvictim @russjones @deusxanima @fheinecke