docker pull does not work for windows when connected to corporate vpn cisco anyconnect

[ripun]

Actual Behavior

We are using Rancher desktop for local development activities , as part of work we need to do "docker push/pull" from our privately hosted container registries

We installed Rancher desktop 1.4.1 on windows and when we connect to corporate VPN then docker pull does not work and fails with below error

docker pull <private_docker_registry>.companycorp.com/os-slim:7
Error response from daemon: Get "https://<private_docker_registry>.companycorp.com/v2/": **dial tcp: lookup** <private_docker_registry>.companycorp.com on xx.xx.xx.xx:53: **server misbehaving**

docker pull <private_docker_registry>.companycorp.com/os:7
Error response from daemon: Get "https://<private_docker_registry>.companycorp.com/v2/": dial tcp: lookup <private_docker_registry>.companycorp.com on xxx.xx.xx.xx:53: read udp xx.xx.xx.xx:5xxx2->xx.xx.xx.xx:53: i/o timeout

Steps to Reproduce

Install Rancher desktop 1.3/1.4.1
Enable Corporate VPN(Cisco anyconnect)
try to docker pull from corporate private docker registry

Result

dial tcp: lookup <private_docker_registry>.companycorp.com on xx.xx.xx.xx:53: server misbehaving

Expected Behavior

It should download/pull image

Additional Information

tried "experimentalHostResolver": false and other few things but nothing helped

Rancher Desktop Version

1.4.1

Rancher Desktop K8s Version

1.24

Which container runtime are you using?

moby (docker cli)

What operating system are you using?

Windows

Operating System / Build Version

Windows 10 Pro

What CPU architecture are you using?

x64

Linux only: what package format did you use to install Rancher Desktop?

No response

Windows User Only

Cisco Anyconnect VPN
This issue should be addressed

[ripun]

@Nino-K @jandubois please advise

[jandubois]

Please try enabling the experimental host resolver and report back if that fixes your problem. This will become the default resolver in the next release.

[ripun]

@jandubois tried setting "experimentalHostResolver": true but still "docker pull " fails when connected to VPN
even nslookup does not work from inside wsl2 for private docker registry FQDN

[jandubois]

Did you stop Rancher Desktop completely (make sure the icon from the system tray is gone), and restart it? The change only takes effect during the start of the VM.

Also double-check that your change is still in the setting file; if you edit it while Rancher Desktop is running, it may get overwritten again because the app doesn't expect the file to be modified at runtime.

[cinnamon-coder-hub]

@ripun Have you tried out wsl-vpnkit. I also had an issue with 'docker pull'. After installing the kit as mentioned on their page, starting Rancher Desktop and afterwards establishing the VPN-connection it worked. Still the order is important in my case. First Rancher Desktop needs to be up (I just wait for the loading bar to disappear). After that the VPN-connection can be established and the command may also work in your case now.
Does this work for you?

[Nino-K]

@cinnamon-coder-hub thanks for the suggestion, however, we have implemented experimentalHostResolver specifically so that our users would not need to rely on a 3rd party wsl packages since they cannot be installed in all corporate environments. The experimentalHostResolver should do the trick when VPN is enabled and you would not need any additional installation of 3rd party patches for WSL. @ripun did the workaround that @jandubois suggested work for you?

[ripun]

We tried already "experimentalHostResolver":true but it does not help

[ripun]

same issue

docker pull <company>.com/os:7-slim
Error response from daemon: Get "https://<company>.com/v2/": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

[Nino-K]

@ripun thanks for the update, I just want to narrow down the issue. Have you tried accessing this private registry prior to using Rancher Desktop (has it ever worked before)? or have you attempted to ping it from the host machine? I just want to make sure that the middle appliances (Cisco ASA Adaptive Security Appliance and WSA Web Security Appliance) that are used by cisco anyconnect are not filtering out your private registry port. You could also try using nmap utility to see if the registry port is potentially being filtered out by the anyconnect infrastructure. nmap -p [REGSITRY_PORT] company.com (might be slightly different on Windows).

Also, have you tried to run docker login to see if that works?

[ripun]

@Nino-K sorry for late reply , Yes same private registry is accessible from MacOS using rancher-desktop
same private registry is accessible from windows with VPN using docker-desktop , but we are unable to connect to same private registry using rancher desktop on windows with VPN.

docker login also fails on windows with VPN for rancher-desktop

[Nino-K]

@ripun Thanks for the update, I got a chance to test this on my side with a similar setup as you described (accessing a private registry via Cisco anyconnect with rancher desktop), and I successfully managed to pull images and ping my private registry.

However, I would like to point out a few things that came to my attention:

1. Cisco anyconnect client settings include an allowlist of all the allowed IP addresses under "Route Details" please make sure your private registry IP address is included in that list. However, this is less likely to be your case since you mentioned that you managed to use the same registry with Docker Desktop in the past.

2. The other thing to look for is any potential IP conflict with the allow list under "Route Details" with the WSL interface IP address or docker IP range. Please make sure there are no IP collisions.

Thanks!

[Nino-K]

@ripun are you able to give this on our new 1.6 release, since I managed to reproduce this myself and found a workaround?

[ripun]

@Nino-K Will ask other team members on windows to check this , I have mac currently

[Nino-K]

@ripun please use our latest release 1.9 which includes a solution for when using Rancher Desktop behind a VPN. You can enable it using:

rdctl set --experimental.virtual-machine.networking-tunnel=true

I'm going to close this issue, feel free to re-open it if the issue still persists.