AWS peer discovery tries to use an IPv4-specific API endpoint in an IPv6-only environment

[ovedeneev]

Describe the bug

IP address 169.254.169.254 is hardcoded in the plugin. Even when server is IPv6-only, plugin still tries this IPv4 instead of IPv6 fd00:ec2::254

...
[debug] <0.253.0> DB: this node is virgin: true
[info] <0.253.0> Peer discovery: configured backend: rabbit_peer_discovery_aws
[debug] <0.253.0> Peer discovery: backend supports initialisation
[debug] <0.253.0> Peer discovery AWS: initialising...
[debug] <0.253.0> HTTP client proxy is not configured
[debug] <0.253.0> Peer discovery: backend initialisation succeeded
[debug] <0.253.0> Peer discovery: registration unsupported, skipping register
[debug] <0.253.0> Feature flags: configure initial feature flags state
[debug] <0.315.0> Feature flags: registering controller globally before proceeding with task: enable_default
[debug] <0.315.0> Feature flags: [global sync] @ rabbit@i-00c377ea0f4e975e1.us-west-1.compute.internal
[debug] <0.315.0> Feature flags: [global register] @ rabbit@i-00c377ea0f4e975e1.us-west-1.compute.internal
[debug] <0.315.0> Feature flags: controller globally registered; can proceed with task
[info] <0.315.0> Feature flags: automatic enablement of feature flags disabled (i.e. none will be enabled automatically)
[debug] <0.315.0> Feature flags: starting an unclustered node for the first time: all feature flags are forcibly left disabled from the $RABBITMQ_FEATURE_FLAGS environment variable
[debug] <0.315.0> Feature flags: unregistering controller globally
[debug] <0.315.0> Feature flags: [global unregister] @ rabbit@i-00c377ea0f4e975e1.us-west-1.compute.internal
[info] <0.253.0> DB: virgin node -> run peer discovery
[debug] <0.253.0> Will use AWS access key of 'ACCESSKEYID'
[debug] <0.253.0> Setting AWS region to "us-west-1"
[debug] <0.253.0> Setting AWS credentials, access key: 'ACCESSKEYID'
[debug] <0.253.0> Querying instance metadata service: "http://169.254.169.254/latest/meta-data/instance-id"
[debug] <0.253.0> EC2 Instance Metadata Service v2 (IMDSv2) is preferred.
[debug] <0.253.0> EC2 IMDSv2 token has not yet been obtained
[info] <0.253.0> Attempting to obtain EC2 IMDSv2 token from "http://169.254.169.254/latest/api/token" ...
[warning] <0.253.0> Failed to obtain EC2 IMDSv2 token: {error,
[warning] <0.253.0>                                     {failed_connect,
[warning] <0.253.0>                                      [{to_address,{"169.254.169.254",80}},
[warning] <0.253.0>                                       {inet,[inet],enetunreach}]}}. Falling back to EC2 IMDSv1 for now. It is recommended to use EC2 IMDSv2. Please refer to the AWS documentation for details on how to configure the instance metadata service: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html.
[debug] <0.253.0> Peer discovery: backend returned the following configuration:
[debug] <0.253.0>   {error,"Failed to determine EC2 instance ID from metadata service"}
[error] <0.253.0> Peer discovery: failed to query the list of nodes from the backend: {error,"Failed to determine EC2 instance ID from metadata service"}
[debug] <0.253.0> Peer discovery: retrying to create/sync cluster in 1000 ms (29 attempts left)
...

HTTP request from shell via IPv6 works fine and does not work via IPv4 which is expected

# docker exec rabbitmq-server wget -qO- http://169.254.169.254/latest/meta-data/instance-id
wget: can't connect to remote host (169.254.169.254): Network unreachable

# docker exec rabbitmq-server wget -qO- http://[fd00:ec2::254]/latest/meta-data/instance-id
i-00c377ea0f4e975e1

Reproduction steps

rabbitmq.conf

loopback_users.guest = false
listeners.tcp.default = 5672
vm_memory_high_watermark.relative = 0.7
cluster_partition_handling = autoheal
tcp_listen_options.keepalive = true
cluster_formation.peer_discovery_backend = aws
prometheus.tcp.port = 15692
prometheus.tcp.ip = ::
management.tcp.ip = ::
management_agent.disable_metrics_collector = false
proxy_protocol = true
log.console = true
log.console.level = debug
log.file = false
channel_max = 0
cluster_formation.aws.use_autoscaling_group = true
cluster_formation.aws.region = us-west-1
cluster_formation.aws.access_key_id = ACCESSKEYID
cluster_formation.aws.secret_key = SECRETKEY
aws.prefer_imdsv2 = false

docker variables

RABBITMQ_SERVER_ADDITIONAL_ERL_ARGS="-kernel inetrc '/etc/rabbitmq/erl_inetrc' -proto_dist inet6_tcp -setcookie COOKIE"
RABBITMQ_CTL_ERL_ARGS="-proto_dist inet6_tcp -setcookie COOKIE"
RABBITMQ_USE_LONGNAME=true
RABBITMQ_IMAGE_VERSION="rabbitmq:3.13.7-alpine"
NODENAME=rabbit@i-00c377ea0f4e975e1.us-west-1.compute.internal

erl_inetrc file
{inet6,true}.

Expected behavior

On IPv6-only server use address fd00:ec2::254 in httpc request perform_http_get_instance_metadata

Additional context

No response

[lukebakken]

@ovedeneev I have a proposed fix here - #14975

If there's any way you can assist with testing it, that would be great, as I don't have an IPv6-only environment. What would you need? A docker image with the fix? I see you're using rabbitmq:3.13.7-alpine.

[ovedeneev]

@lukebakken I'm happy to assist with testing the fix, docker image would be great.

Thank you.

[michaelklishin]

@ovedeneev I hope you are aware of the fact that RabbitMQ 3.13 is out of community support and will not receive any new public releases.
Consider migrating away from 3.x ASAP.

There is a workflow that produces an OCI image for every PR but it is excluded for "external contributors" and that includes @lukebakken's PRs.

[michaelklishin]

This action run will produce a new OCI off of main that will include #14975.

The new tag should show up on Docker Hub in under 10 minutes.

[michaelklishin]

This tag should include #14975.

Once confirmed effective, we can backport that change to v4.2.x and v4.1.x.

[michaelklishin]

@ovedeneev also, I cannot help but call out this line:

loopback_users.guest = false

Enabling remote access for the default user with publicly known credentials is one of the worst practices. The docs very explicitly state so:

The recommended way to address this in production systems is to create
a new user with generated credentials, or set of users,
with the permissions to access the necessary virtual hosts.

Importing a definitions file with some virtual hosts, users and permissions is not particularly difficult, whether RabbitMQ is deployed in a container a not. Just mount a file and point the node at it.

[lukebakken]

@ovedeneev please give this docker image a try in your IPv6-only environment. Thanks.