IPv6 only works when network is host driver, and does not work when network is bridge driver

[Motophan]

• This is a bug report
• I searched existing issues before opening this one
  Similar issue Incorrect IPV6 prefix address assigned to container interface #931
  In my situation the IPv6 addresses that are being assigned to the container are technically correct, but do not have proper connectivity. I believe I need a separate issue because of this.
• This is a feature request

cat /etc/docker/daemon.json

    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}
{
  "ipv6": true,
  "fixed-cidr-v6": "2X05:7280:6::2/64"
}

cat /etc/netplan/50-cloud-init.yaml

# This file is generated from information provided by
# the datasource.  Changes to it will not persist across an instance.
# To disable cloud-init's network configuration capabilities, write a file
# /etc/cloud/cloud.cfg.d/99-disable-network-config.cfg with the following:
# network: {config: disabled}
network:
  version: 2
  renderer: networkd

  ethernets:
    enp134s0f1:
      dhcp4: false
      dhcp6: false

  bridges:
    br0:
      interfaces:
        - enp134s0f1
      addresses:
        - 104.XXX.196.169/27
        - 
      gateway4: 104.XXX.196.190
      gateway6: "2X05:7280:6::1"
      nameservers:
        addresses:
          - 1.1.1.1
          - 1.0.0.1
          - 2606:4700:4700::1111
          - 2606:4700:4700::1001

X has replaced a fully provisioned IP space, redacted to comply with github community guidelines.

Expected behavior

Containers aware of external IPv6 space, and ICBM ping would work.

Actual behavior

Network unreachable.

Known information:
1- Host can ping ipv6 space
2- Container on host network can ping ipv6 space
3- Container on bridge defined network (so all my containers can talk to eachother and the outside world) cannot ping ipv6 space

ping ipv6.google.com
PING ipv6.google.com(sfo07s26-in-x0e.1e100.net (2607:f8b0:4005:802::200e)) 56 data bytes
64 bytes from sfo07s26-in-x0e.1e100.net (2607:f8b0:4005:802::200e): icmp_seq=1 ttl=55 time=1.47 ms
64 bytes from sfo07s26-in-x0e.1e100.net (2607:f8b0:4005:802::200e): icmp_seq=2 ttl=55 time=1.52 ms

docker run -it --rm busybox ping ipv6.google.com
PING ipv6.google.com (2607:f8b0:4005:802::200e): 56 data bytes
ping: sendto: Cannot assign requested address

Now using host driver

docker run -it --rm --network=host busybox ping ipv6.google.com
PING ipv6.google.com (2607:f8b0:4005:802::200e): 56 data bytes
64 bytes from 2607:f8b0:4005:802::200e: seq=0 ttl=55 time=1.740 ms
64 bytes from 2607:f8b0:4005:802::200e: seq=1 ttl=55 time=1.600 ms

attempt to make a ipv6 aware bridge (see cat of /etc/docker/daemon.json above)

sudo docker network create --ipv6 ipv6test
Error response from daemon: could not find an available, non-overlapping IPv6 address pool among the defaults to assign to the network

note, following command is redacted, unredacted command sent to daemon and returned without error.

sudo docker network create --ipv6 --gateway="2X05:7280:6::1" --subnet="2X05:7280:6::2/64" ipv6test
e33cd837d7ba91fdc112b0cd76c4bb773d01a591657be51c0886401e6c645adc

sudo docker network inspect ipv6test
[
{
"Name": "ipv6test",
"Id": "0228670d15bd3c5063103fe2840cbb83d9741b837dfeeaef1ba0d3936603f102",
"Created": "2020-06-01T20:29:09.254618852-07:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": true,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.23.0.0/16",
"Gateway": "172.23.0.1"
},
{
"Subnet": "2X05:7280:6::2/64",
"Gateway": "2X05:7280:6::1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {},
"Options": {},
"Labels": {}
}
]

docker run -it --rm --network=ipv6test busybox ping ipv6.google.com
PING ipv6.google.com (2607:f8b0:4005:802::200e): 56 data bytes
^C
--- ipv6.google.com ping statistics ---
7 packets transmitted, 0 packets received, 100% packet loss

Steps to reproduce the behavior

1- Provision IPv6 addresses
2- install docker
3- attempt to make a ipv6 aware network
4- attempt to ping ipv6 only space.

Output of docker version:

Client: Docker Engine - Community
 Version:           19.03.8
 API version:       1.40
 Go version:        go1.12.17
 Git commit:        afacb8b7f0
 Built:             Wed Mar 11 01:25:46 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.8
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.12.17
  Git commit:       afacb8b7f0
  Built:            Wed Mar 11 01:24:19 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Output of docker info:

Client:
 Debug Mode: false

Server:
 Containers: 22
  Running: 22
  Paused: 0
  Stopped: 0
 Images: 121
 Server Version: 19.03.8
 Storage Driver: overlay2
  Backing Filesystem: <unknown>
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: nvidia runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.1.21-050121-generic
 Operating System: Ubuntu 18.04.4 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.81GiB
 Name: systemdnet
 ID: V7ZF:SDIY:SLQS:7SAK:TAMH:G77I:ZNBS:XQBB:YGVR:EUJD:VP2X:2UJN
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.)
Reproduced on two seperate dedicated server hosts, though both were of similar host operating systems (ubuntu and debian).

I can reproduce this on a hetzner vps, and can send the login details to a dev if needed. No confidential material exists on the vps and they provision their vps's with a /64 ipv6 space.

[h1ght]

on my local machine at home, using debian 12 repo, i can route ipv6 with prefix delegation. ip6tables blocks icmp pings to the containers, but i can allow this. forwarding rules are created, too.

Client:
 Version:           20.10.24+dfsg1
 API version:       1.41
 Go version:        go1.19.8
 Git commit:        297e128
 Built:             Thu May 18 08:38:34 2023
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.24+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.19.8
  Git commit:       5d6db84
  Built:            Thu May 18 08:38:34 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.20~ds1
  GitCommit:        1.6.20~ds1-1+b1
 runc:
  Version:          1.1.5+ds1
  GitCommit:        1.1.5+ds1-1+b1
 docker-init:
  Version:          0.19.0
  GitCommit:

even ports are opened via ip6tables.

Chain DOCKER (2 references)
target     prot opt source               destination
ACCEPT     tcp  --  anywhere             fe80::42:242:ac11:2  tcp dpt:7474
ACCEPT     tcp  --  anywhere             2a00:6xxx8:20fc::2  tcp dpt:9696
ACCEPT     tcp  --  anywhere             fe80::42:242:ac11:3  tcp dpt:9443
ACCEPT     tcp  --  anywhere             2a00:60xxx0fc::3  tcp dpt:7878
ACCEPT     tcp  --  anywhere             fe80::42:242:ac11:3  tcp dpt:8000
ACCEPT     tcp  --  anywhere             2a00:60xxx8:20fc::4  tcp dpt:8989
ACCEPT     tcp  --  anywhere             2a00:6xx0fc::6  tcp dpt:http-alt
ACCEPT     tcp  --  anywhere             2a00:60xx28:20fc::6  tcp dpt:https
ACCEPT     tcp  --  anywhere             2a00:60xxx28:20fc::6  tcp dpt:http

try smaller subnet, like ipv6/80 and add a whole :\abcd: group e.g.
subnet
"2X05:7280:6:BEEF::/80"
Gateway
"2X05:7280:6:BEEF::1"
this worked for me with
dont forget to allow ipv6 forwarding and maybe at a static route via ip -6 route add "2X05:7280:6:BEEF::/80" dev (bridge interface)

Client: Docker Engine - Community
 Version:           24.0.7
 API version:       1.43
 Go version:        go1.20.10
 Git commit:        afdd53b
 Built:             Thu Oct 26 09:08:02 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.7
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.10
  Git commit:       311b9ff
  Built:            Thu Oct 26 09:08:02 2023
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.24
  GitCommit:        61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
 runc:
  Version:          1.1.9
  GitCommit:        v1.1.9-0-gccaecfc
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

but with the docker repo version i cannot change default bridge cidr in /etc/docker/daemon.json anymore. after restart it fails and im stuck with the existing cidr in the default bridge.