ubuntu-latest 20240609.1.0 is forcing IPv6 on Verdaccio

[cortinico]

Description

Hey all,
@cortinico from React Native here.

I've noticed that the latest deployment of the ubuntu-latest runner on GitHub actions (version 20240609.1.0) is breaking our CI on React Native causing it to take 6+ hours to complete.

This happend as the runner was updated from 20240603.1.0 to 20240609.1.0
(see here https://github.com/actions/runner-images/releases/tag/ubuntu22%2F20240609.1).

✅ Successful Run: https://github.com/facebook/react-native/actions/runs/9473329852/job/26101842381
❌ Faulty Run (6h+): https://github.com/facebook/react-native/actions/runs/9484523671/job/26136577410

The problem seems to be related to IPv4 vs IPv6.
We do use Verdaccio in our CI setup to test NPM packages. For some reasons, Verdaccio is always listening on a IPv6 port since 20240609.1.0.

I'm unsure what could be causing this, but I tried all possible ways to force Verdaccio to listen on a IPv4 address/port. It seems like to always fallback to an IPv6.

Happy to have more insights on what caused this as nothing changed other than the Ubuntu Runner version.

I also created a reproducer here:
https://github.com/cortinico/repro-ipv6-gha-issue/

You can see in the latest run how I'm running Verdaccio and using net-stat to show the open ports:
https://github.com/cortinico/repro-ipv6-gha-issue/actions/runs/9503903039/job/26195373761
Seems like Verdaccio is listening on a IPv6 port:

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 11
• macOS 12
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• Windows Server 2019
• Windows Server 2022

Image version and build link

20240609.1.0

Is it regression?

20240603.1.0

Expected behavior

Processes should honor the port/address configuration provided

Actual behavior

Process is ignoring the configuration and always opening an IPv6 port/address.

Repro steps

Reproducer is here:
https://github.com/cortinico/repro-ipv6-gha-issue/

[ArminArdekani]

Surprised this hasn't been flagged as a incident yet by Github Actions, considering ipv6 isn't supported. It's affecting our CI jobs today too.

The culprit is when they updated the runners to use Docker v26 which has the loopback: https://docs.docker.com/engine/release-notes/26.0/#bug-fixes-and-enhancements-2

[cortinico]

@lakshminarayana02 @vidyasagarnimmagaddi any news on what's the ETA for this fix? Also are there any workaround?

[cortinico]

For context this is broken among ubuntu-20.04, ubuntu-latest and ubuntu-24.04

[vidyasagarnimmagaddi]

Hi @cortinico , we are analyzing the issue, will revert ASAP

[vidyasagarnimmagaddi]

Hi @cortinico , We tried reproducing the same.In file verdaccio.yml , add
listen:

• 0.0.0.0:4873

kindly revert if any issues.

[cortinico]

We tried reproducing the same.In file verdaccio.yml , add
listen:

Hi @vidyasagarnimmagaddi Thanks for getting back to me.
I'm not sure I fully understand your answer.

As you see from my reproducer, when calling netstat -ano the open port still shows up as IPv6.

Moreover it seems also other users had similar issues (@ansgarm and @ArminArdekani)

[erik-bershel]

Hey @cortinico!

Please use a workaround config for the tool with using of IPv4 address directly. For example:

listen:
   0.0.0.0:4873

The same time we'll investigate why the default resolver for localhost was changed to IPv6 instead of IPv4.

[ansgarm]

Hi everyone 👋

I can confirm that explicitly configuring an IPv4 address worked for us 👍 Thanks for the workaround instructions, @erik-bershel!

[cortinico]

Thanks for the clarification @erik-bershel 👍

[erik-bershel]

Hey @cortinico

After further investigation, I can say the following. Correct me if you see any discrepancies.

• Your tests run in a container
• After updating Docker, the tool began to create a listener on the local IPv6 address

In general, the latest Docker changes have actually caught up with similar changes that have existed for a long time, including on our Ubuntu runners. If you try to run the ping localhost command you will find that it accesses the address ::1. In general, the problem is not even related to the fact that Docker added the localhost alias to the ::1 address, but to the fact that by default Ubuntu and many other systems consider IPv6 addresses to have higher priority when resolving names.

You have several ways to work around this problem in Docker:

• you can use IPv4 addresses instead of the localhost alias, as I recommended earlier for this particular tool and not only
• you can lower the priority of IPv6 addresses by making changes to the /etc/gai.conf file, there should be enough to add the line "precedence ::ffff:0:0/96 100"
• you can edit the /etc/hosts file and replace the line ::1 localhost ip6-localhost ip6-loopback with ::1 ip6-localhost ip6-loopback
• you can try using the option -sysctl=net.ipv6.conf.all.disable_ipv6=1 when starting the container, but according to the ubuntu-22.04 / 20240609.1.0 issues with IPv6 DNS resolution inside container #10088 it might not help

Unfortunately, Docker updates constantly bring new calls, yeah. But right now we would not want to roll back these changes for a number of reasons:

• a fairly large gap has accumulated between the current state in the images and the latest available tools
• existing problem might be circumvented at runtime
• the investigation is still ongoing

cc @ansgarm @ArminArdekani @samjarrett

[ArminArdekani]

@erik-bershel Our problem from this was that we could not communicate with the docker containers HEALTHCHECK endpoint that was specified in the Dockerfile with localhost. So we managed to fix it by replacing localhost with 0.0.0.0 address in the dockerfile.

[cortinico]

You have several ways to work around this problem in Docker:

Thanks for the workaround @erik-bershel 👍 this unblocks us

[rogertinsley]

Had the same issue with maildev with the healthcheck in the Dockerfile:

https://github.com/maildev/maildev/blob/357a20edcd205413d3590aedb8fcd7c97563c40d/Dockerfile#L23

Ended up adding an environment variable to unblock me:

  maildev:
    hostname: maildev
    image: maildev/maildev
    ports:
      - 1080:1080
      - 1025:1025
    environment:
      MAILDEV_IP: '::'

[vidyasagarnimmagaddi]

Hi @cortinico , hope your issue got resolved, Closing this issue. Thanks