Using IMDS.get in a Docker container fails to fall back to IMDSv1

[omus]

When running within a Docker container within an EC2 instance the IMDS client attempts to use IMDSv2 which fails to request a session token and then fails to fall back to IMDSv1. Can be reproduced by running a Docker container within an EC2 instance:

docker run -it julia:1.8.5-buster

julia> using Pkg; Pkg.add(PackageSpec(name="AWS", version="1.90.2"))

julia> using AWS: IMDS

julia> session = IMDS.Session("", 0, typemax(Int64)); IMDS.request(session, "GET", "/latest/meta-data/iam/info")  # IMDSv1
HTTP.Messages.Response:
"""
HTTP/1.1 200 OK
Content-Type: text/plain
Accept-Ranges: none
Last-Modified: Tue, 01 Aug 2023 13:24:21 GMT
Content-Length: 201
Date: Tue, 01 Aug 2023 14:20:32 GMT
Server: EC2ws
Connection: close

{
  "Code" : "Success",
  "LastUpdated" : "2023-08-01T13:24:21Z",
  "InstanceProfileArn" : "arn:aws:iam::...",
  "InstanceProfileId" : "..."
}"""

julia> session = IMDS.Session(); IMDS.request(session, "GET", "/latest/meta-data/iam/info")  # IMDSv2 with fallback to IMDSv1
ERROR: AWS.AWSExceptions.IMDSUnavailable: The Instance Metadata Service is unavailable on the host

Stacktrace:
 [1] _http_request(::String, ::Vararg{Any}; status_exception::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:106
 [2] refresh_token!(session::AWS.IMDS.Session, duration::Int16)
   @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:59
 [3] refresh_token!
   @ ~/.julia/packages/AWS/1nULH/src/IMDS.jl:52 [inlined]
 [4] request(session::AWS.IMDS.Session, method::String, path::String; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:83
 [5] request(session::AWS.IMDS.Session, method::String, path::String)
   @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:79
 [6] top-level scope
   @ REPL[11]:1

caused by: HTTP.RequestError:
HTTP.Request:
HTTP.Messages.Request:
"""
PUT /latest/api/token HTTP/1.1
X-aws-ec2-metadata-token-ttl-seconds: 600
Host: 169.254.169.254
Accept: */*
User-Agent: HTTP.jl/1.8.5
Content-Length: 0
Accept-Encoding: gzip

"""Underlying error:
IOError: read: connection timed out (ETIMEDOUT)
Stacktrace:
  [1] (::HTTP.ConnectionRequest.var"#connections#4"{HTTP.ConnectionRequest.var"#connections#1#5"{HTTP.TimeoutRequest.var"#timeouts#3"{HTTP.TimeoutRequest.var"#timeouts#1#4"{HTTP.ExceptionRequest.var"#exceptions#2"{HTTP.ExceptionRequest.var"#exceptions#1#3"{typeof(HTTP.StreamRequest.streamlayer)}}}}}})(req::HTTP.Messages.Request; proxy::Nothing, socket_type::Type, socket_type_tls::Type, readtimeout::Int64, connect_timeout::Int64, logerrors::Bool, logtag::Nothing, kw::Base.Pairs{Symbol, Union{Nothing, Integer}, NTuple{4, Symbol}, NamedTuple{(:iofunction, :decompress, :verbose, :status_exception), Tuple{Nothing, Nothing, Int64, Bool}}})
    @ HTTP.ConnectionRequest ~/.julia/packages/HTTP/nn2yB/src/clientlayers/ConnectionRequest.jl:143
...
  [8] #request#19
    @ ~/.julia/packages/HTTP/nn2yB/src/HTTP.jl:315 [inlined]
  [9] macro expansion
    @ ~/.julia/packages/Mocking/Q17aB/src/mock.jl:29 [inlined]
 [10] _http_request(::String, ::Vararg{Any}; status_exception::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:97
 [11] refresh_token!(session::AWS.IMDS.Session, duration::Int16)
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:59
 [12] refresh_token!
    @ ~/.julia/packages/AWS/1nULH/src/IMDS.jl:52 [inlined]
 [13] request(session::AWS.IMDS.Session, method::String, path::String; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:83
 [14] request(session::AWS.IMDS.Session, method::String, path::String)
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:79
 [15] top-level scope
    @ REPL[11]:1

caused by: TaskFailedException

    nested task error: IOError: read: connection timed out (ETIMEDOUT)
    Stacktrace:
     [1] wait_readnb(x::Sockets.TCPSocket, nb::Int64)
       @ Base ./stream.jl:410
     [2] eof(s::Sockets.TCPSocket)
       @ Base ./stream.jl:106
     [3] read_to_buffer(c::HTTP.Connections.Connection{Sockets.TCPSocket}, sizehint::Int64)
       @ HTTP.Connections ~/.julia/packages/HTTP/nn2yB/src/Connections.jl:220
     [4] readuntil(c::HTTP.Connections.Connection{Sockets.TCPSocket}, f::typeof(HTTP.Parsers.find_end_of_header), sizehint::Int64)
       @ HTTP.Connections ~/.julia/packages/HTTP/nn2yB/src/Connections.jl:240
     [5] readuntil
       @ ~/.julia/packages/HTTP/nn2yB/src/Connections.jl:238 [inlined]
     [6] readheaders
       @ ~/.julia/packages/HTTP/nn2yB/src/Messages.jl:533 [inlined]
     [7] startread(http::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.Connections.Connection{Sockets.TCPSocket}})
       @ HTTP.Streams ~/.julia/packages/HTTP/nn2yB/src/Streams.jl:153
     [8] macro expansion
       @ ~/.julia/packages/HTTP/nn2yB/src/clientlayers/StreamRequest.jl:51 [inlined]
     [9] (::HTTP.StreamRequest.var"#3#5"{Nothing, HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.Connections.Connection{Sockets.TCPSocket}}, HTTP.Messages.Request, HTTP.Messages.Response, Float64, ReentrantLock})()
       @ HTTP.StreamRequest ./threadingconstructs.jl:258
Stacktrace:
  [1] sync_end(c::Channel{Any})
    @ Base ./task.jl:436
  [2] macro expansion
    @ ./task.jl:455 [inlined]
  [3] streamlayer(stream::HTTP.Streams.Stream{HTTP.Messages.Response, HTTP.Connections.Connection{Sockets.TCPSocket}}; iofunction::Nothing, decompress::Nothing, logerrors::Bool, logtag::Nothing, timedout::Nothing, kw::Base.Pairs{Symbol, Int64, Tuple{Symbol}, NamedTuple{(:verbose,), Tuple{Int64}}})
    @ HTTP.StreamRequest ~/.julia/packages/HTTP/nn2yB/src/clientlayers/StreamRequest.jl:34
... [13] #request#19
    @ ~/.julia/packages/HTTP/nn2yB/src/HTTP.jl:315 [inlined]
 [14] macro expansion
    @ ~/.julia/packages/Mocking/Q17aB/src/mock.jl:29 [inlined]
 [15] _http_request(::String, ::Vararg{Any}; status_exception::Bool, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:97
 [16] refresh_token!(session::AWS.IMDS.Session, duration::Int16)
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:59
 [17] refresh_token!
    @ ~/.julia/packages/AWS/1nULH/src/IMDS.jl:52 [inlined]
 [18] request(session::AWS.IMDS.Session, method::String, path::String; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:83
 [19] request(session::AWS.IMDS.Session, method::String, path::String)
    @ AWS.IMDS ~/.julia/packages/AWS/1nULH/src/IMDS.jl:79
 [20] top-level scope
    @ REPL[11]:1

This issue does not occur when running directly on the EC2 instance itself and the issue in Docker can be corrected by increasing the hop limit to 2. However, the AWS.jl code should automatically fall back to using IMDSv1 in this scenario if it is available.

[omus]

Additional details on the hop limit can be found under the "Protecting against open layer 3 firewalls and NATs" section:
https://aws.amazon.com/blogs/security/defense-in-depth-open-firewalls-reverse-proxies-ssrf-vulnerabilities-ec2-instance-metadata-service/