1.15.0 IAM auth broken - invalid memory addr or nil pointer deref

[rjhornsby]

Describe the bug

Running vault servers on Rocky 8 x64, AWS ec2 instances.

There's an issue in vault server 1.15.0 not present in versions <= 1.14.4 where vault clients/agents attempting to use aws (IAM) auth get back a 502 error from the vault server.

The server logs provide some additional information:

Oct 11 17:48:27 vault-05ee13 vault[4130976]: 2023-10-11T17:48:27.095-0400 [ERROR] core: panic serving forwarded request: path=/v1/auth/aws/mycorp/myunit/login error="runtime error: invalid memory address or nil pointer dereference"
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  stacktrace=
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | goroutine 410065 [running]:
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | runtime/debug.Stack()
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | \t/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/debug/stack.go:24 +0x5e
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | github.com/hashicorp/vault/vault.(*forwardedRequestRPCServer).ForwardRequest.func1.1()
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | \t/home/runner/work/vault/vault/vault/request_forwarding_rpc.go:48 +0x46
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | panic({0x840f080?, 0x10f040e0?})
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | \t/opt/hostedtoolcache/go/1.21.1/x64/src/runtime/panic.go:914 +0x21f
Oct 11 17:48:27 vault-05ee13 vault[4130976]:  | github.com/hashicorp/vault/builtin/credential/aws.(*backend).pathLoginIamGetRoleNameCallerIdAndEntity(0xc0040f7800, {0xb8b5968, 0xc003a50d80}, 0xc004260e00, 0x7ba56c0?)
...

The stacktrace is very long, so the remainder of it is in a gist[1]. For now I've rolled our servers back to 1.14.x, while the clients/agents seem to be perfectly happy to continue on 1.15.0.

[1] https://gist.github.com/rjhornsby/1ec8a40fa9ec5d44376ad5b8bc540ba1

To Reproduce
Steps to reproduce the behavior:

1. Upgrade vault server to 1.15.0
2. Attempt to use AWS (IAM) authentication from a client/agent against this server.

Sample ruby snippet:

credential_provider = Aws::InstanceProfileCredentials.new(region: m.get('/latest/meta-data/placement/region'))
client = Vault::Client.new(address:'https://vault.mycorp.com:8200')
client.auth.aws_iam('a-valid-role', credential_provider, nil, 'https://sts.amazonaws.com','/v1/auth/aws/mycorp/myunit/login')

Expected behavior
As per the behavior of 1.14.x, the server should perform the appropriate authentication and return credentials to the caller.

Environment:

• Vault Server Version: 1.15.0
• Vault CLI Version: 1.15.0
• Server Operating System/Architecture: Rocky 8.8 / x64

Additional context
As noted above, rolling the servers back to 1.14.x resolves this issue. There are a bunch of layers involved here, not the least of which is how the AWS API responds to requests from the server, making it difficult to know what details would be helpful for diagnostics.

[rjhornsby]

re the agent label, it is the agent - but it's also not the agent. The problem appears to be more specifically in the server. The agent forwards its request to the server, which tries to contact AWS on behalf of the agent (or any other vault client AFAICT). It's in the server where the failure (and the associated stacktrace) happens.

[raskchanky]

@rjhornsby Thanks for the bug report! It looks like this was fixed here #23555 and should be released in 1.15.1.