Podman pull fails due to Docker Hub rate limit with IPv6 addressing

[Amygos]

An issue has been identified where Docker Hub rate limits are enforced based on IPv6 addresses, leading to failed image pulls on systems sharing the same IPv6 subnet. This behavior affects all images hosted on Docker Hub and occurs during updates or installations.

Observations

• The docker-ratelimit-source header indicates that rate limits are applied based on the shared IPv6 prefix:

  docker-ratelimit-source: 2604:a880:400:d0::
  

• Multiple VPSs within the same subnet share this IPv6 prefix, causing the rate limit to be reached quickly.

Modules and Images Involved

The following command was used to identify modules and images pulled from Docker Hub:

curl https://raw.githubusercontent.com/NethServer/ns8-repomd/refs/heads/repomd/repodata.json | \
jq -r '.[] | .name as $name | .versions[] | .tag as $tag | .labels."org.nethserver.images" | split(" ") | .[] | select(startswith("docker.io")) | [$name, $tag, .] | @tsv'

The output reveals modules that are potentially affected:

traefik    2.2.5    docker.io/traefik:v2.11.5
ldapproxy    1.1.0    docker.io/library/nginx:1.27.2-alpine
NethVoice    1.1.1-testing.5    docker.io/library/redis:7.0.10-alpine
NethVoice    1.1.0    docker.io/library/redis:7.0.10-alpine
Piler    1.0.0    docker.io/sutoj/piler:1.4.4
Piler    1.0.0    docker.io/mariadb:10.11.7
Piler    1.0.0    docker.io/memcached:1.6.26-alpine
Ejabberd    1.0.7    docker.io/ejabberd/ecs:24.07
CrowdSec    1.0.11    docker.io/crowdsecurity/crowdsec:v1.6.3-debian
Nextcloud    1.1.8    docker.io/redis:6.2.12-alpine
Nextcloud    1.1.8    docker.io/mariadb:10.6.19
Nextcloud    1.1.8    docker.io/nginx:1.27.1-alpine
Roundcubemail    2.0.4    docker.io/mariadb:10.11.9
Roundcubemail    2.0.4    docker.io/roundcube/roundcubemail:1.6.9-apache
Netdata    1.1.2    docker.io/netdata/netdata:v1.47.5
loki    1.2.2    docker.io/traefik:v2.11.13
loki    1.2.2    docker.io/grafana/loki:2.9.10
Nethsecurity Controller    1.1.1-testing.1    docker.io/grafana/promtail:2.7.1
Nethsecurity Controller    1.1.1-testing.1    docker.io/grafana/loki:2.9.4
Nethsecurity Controller    1.1.1-testing.1    docker.io/prom/prometheus:v2.50.1
Nethsecurity Controller    1.1.1-testing.1    docker.io/grafana/grafana:11.2.0
Nethsecurity Controller    1.1.1-testing.1    docker.io/timescale/timescaledb:2.16.1-pg16
Nethsecurity Controller    1.1.0    docker.io/grafana/promtail:2.7.1
Nethsecurity Controller    1.1.0    docker.io/grafana/loki:2.9.4
Nethsecurity Controller    1.1.0    docker.io/prom/prometheus:v2.50.1
Nethsecurity Controller    1.1.0    docker.io/grafana/grafana:11.2.0
Nethsecurity Controller    1.1.0    docker.io/timescale/timescaledb:2.16.1-pg16
core    2.9.6    docker.io/grafana/promtail:2.9.2
core    3.2.3-dev.2    docker.io/grafana/promtail:2.9.2
core    3.2.2    docker.io/grafana/promtail:2.9.2
Mattermost    2.1.0    docker.io/postgres:13.13-alpine
Mattermost    2.1.0    docker.io/mattermost/mattermost-team-edition:9.11.2  

This highlights various modules relying on Docker Hub images, all potentially impacted by the IPv6-based rate limits.

Workaround

Temporarily disabling IPv6 during updates or installations can mitigate the issue:

firewall-cmd --direct --add-rule ipv6 filter OUTPUT 0 -j REJECT  

IPv6 can be re-enabled after the operation is complete.

Suggested Permanent Solutions

1. Download required images from docker.io during the build process to avoid dependency during installations.
2. Set up a private registry with proxying and mirroring capabilities to manage image pulls and reduce reliance on Docker Hub.
3. Utilize remapping and mirroring registries to redirect Docker Hub requests to an alternative or mirrored registry.
4. Use Docker Hub authentication to increase rate limits.

See also

Discussion https://mattermost.nethesis.it/nethesis/pl/izj8uezi7bguzb7yoxsdixgb4c

[DavidePrincipi]

Solution 3 is easy and cheap to implement, with no impact on existing apps. The linked PRs leverage an established Podman feature and GitHub Packages to mirror some docker.io images transparently.

Tom Sweeney and Ed Santiago used the remapping to develop a creative solution to address the rate limits of Docker Hub. In late November 2020, Docker Hub started to limit the number of pulls per user in a given timeframe. At first, we were concerned because large parts of our testing systems, and continuous integration used Docker Hub images. But with a simple change to the registries.conf on our systems, Tom and Ed found a great solution. That spared us from the manual and tedious task of changing all images referring to docker.io in our tests. -- https://www.redhat.com/en/blog/manage-container-registries

Other interestingn articles

• https://www.redhat.com/en/blog/manage-container-registries
• https://www.redhat.com/en/blog/7-transports-features

[DavidePrincipi]

Test case

1. Install core 3.2.3-dev.3. Traefik, Promtail, Loki services (downloaded from docker.io mirror) must be correctly started. Automated test cases should already do this job.
2. With the installation of this update, docker.io rate limit errors should not hit anymore, at least for applications of the default repository. In other cases a docker.io auth token is still necessary https://docs.nethserver.org/projects/ns8/en/latest/dockerhub.html

[lucagasparini]

Test case: VERIFIED

[DavidePrincipi]

Released in https://github.com/NethServer/ns8-core/releases/tag/3.2.3