Connectivity issue after php:8.0-fpm

[mohammedx3]

Description

We are running our dev environment with devspace on k3d cluster, and all services could reach each other on our domains (pod1.example.com) and (pod2.example.com), etc ... until we decided to upgrade one image from php:8.0-fpm to php:8.3-fpm, now the pod having the newer image cannot access other services anymore.
outputs on php:8.3-fpm:

curl -I pod1.example.com
curl: (6) Could not resolve host: pod1.example.com

telnet pod1.example.com 80
Server lookup failure: pod1.example.com, Name or service not known

nslookup pod1.example.com
Server:		10.43.0.10
Address:	10.43.0.10#53

Name:	nginx.all.svc.cluster.local
Address: 10.43.113.49

outputs on php:8.0-fpm:

telnet pod1.example.com 80
Trying 10.43.113.49...
Connected to nginx.all.svc.cluster.local.
Escape character is '^]'.
\q
curl -I pod1.example.com
HTTP/1.1 200 OK

Outside our dev environment, running the docker image locally, curl seems to have different outputs for our domain:
php:8.3-fpm:

curl -I example.com
curl: (7) Failed to connect to example.com port 80 after 70 ms: Couldn't connect to server

php:8.0-fpm:

curl -I example.com
curl: (7) Failed to connect to example.com port 80: Connection refused

We've encountered a connectivity issue that arises specifically when using PHP-FPM versions greater than 8.0, excluding the bullseye images for versions 8.1 and above which works fine. Although DNS resolutions via nslookup is successful, attempts to connect to services using curl or telnet on port 80 fail. This problem is not present in PHP-FPM version 8.0 or earlier, nor does it occur in the bullseye images of PHP-FPM versions 8.1 and above.

Context (Environment)

• devspace (6.3.2) running on k3d (v5.5.1).
• Node version: v1.26.4+k3s1

[yosifkit]

I guess that this is likely due to seccomp -- so, need to update Docker, runc, and libseccomp on your host. You can verify that it is libseccomp by temporarily running the Bookworm-based image with --security-opt seccomp=unconfined, if it works fine, then it is libseccomp related and so updates should be applied to the host.

docker-library/python#837 (comment)

[mohammedx3]

I guess that this is likely due to seccomp -- so, need to update Docker, runc, and libseccomp on your host. You can verify that it is libseccomp by temporarily running the Bookworm-based image with --security-opt seccomp=unconfined, if it works fine, then it is libseccomp related and so updates should be applied to the host.

docker-library/python#837 (comment)

Thanks for the reply.
I tried your solution but still having the same issue with --security-opt seccomp=unconfined
I even updated runc, Docker and libsecomp and trying the run the image locally, still getting the same result.

curl -I example.com
curl: (7) Failed to connect to example.com port 80 after 118 ms: Couldn't connect to server

runc --version
runc version 1.1.11
commit: v1.1.11-0-g4bccb38
spec: 1.0.2-dev
go: go1.20.13
libseccomp: 2.5.5

docker version
Client: Docker Engine - Community
 Version:           25.0.1
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        29cf629
 Built:             Tue Jan 23 23:09:25 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.1
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       71fa3ab
  Built:            Tue Jan 23 23:09:25 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.27
  GitCommit:        a1496014c916f9e62104b33d1bb5bd03b0858e59
 runc:
  Version:          1.1.11
  GitCommit:        v1.1.11-0-g4bccb38
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[tianon]

Did anything about your k3d change recently too?
(This feels reminiscent of docker-library/docker#467, where nftables vs iptables caused similar connectivity issues)

[mohammedx3]

Not really, the problem is happening even outside k3d, I cannot curl our domain while running php:8.3-fpm local docker image but I can do so with the old ones, I can however nslookup it on both.

I even updated k3d version to latest.

[yosifkit]

Does php:8.3-fpm-bullseye work? (instead of the default, which is Debian Bookworm-based)

[mohammedx3]

Does php:8.3-fpm-bullseye work? (instead of the default, which is Debian Bookworm-based)

Yes, it works.

[mohammedx3]

One key thing I just noticed, is that the issue is not with the domain at all, its with what it points to. our domain example.com points to loopback address 127.0.0.1, so you can achieve the same results trying to access this address from the image locally.
is there a reason why the curling 127.0.0.1 from php:8.3-fpm can't reach the host but bullseye image can?

[tianon]

Oh that's strange -- 127.0.0.1 will connect to loopback inside the container (not the host), regardless of which base image you're from. 🤔

[tianon]

I guess the caveat to that is that in k8s, that's shared across a whole pod -- are you running multiple containers in your pods? Is there any reason that the bookworm tests might not be running one of the containers in the pod as the bullseye tests?

[mohammedx3]

maybe I was not clear, I didn't mean the host as in the host machine, I meant the server, curling 127.0.0.1 from bullseye is able to reach the server but the request is refused because nothing is running on port 80.
In non-bullseye images, its unable to reach the server.
bullseye:

docker run -it  --entrypoint=/bin/bash php:8.3-fpm-bullseye
root@016f4009bd89:/var/www/html# curl -I 127.0.0.1
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

non-bullseye:

docker run -it  --entrypoint=/bin/bash php:8.3-fpm
root@1f572d3ee779:/var/www/html# curl -I 127.0.0.1
curl: (7) Failed to connect to 127.0.0.1 port 80 after 0 ms: Couldn't connect to server

[LaurentGoderre]

Fpm isn't like a regular server and doesn't use port 80 or http. Also, by overriding the entrypoint the fpm process doesn't get started. The following works!

docker run -it --rm --name fpm-test -d php:8.3-fpm-bullseye
docker exec -it fpm-test bash

apt update && apt install -y libfcgi0ldbl
cgi-fcgi -bind -connect 127.0.0.1:9000

[mohammedx3]

I dont need to override the entrypoint to get the same outputs

docker run -it --rm --name fpm-test -d php:8.3-fpm
docker exec -it fpm-test bash
curl -I 127.0.0.1
curl: (7) Failed to connect to 127.0.0.1 port 80 after 0 ms: Couldn't connect to server

docker run -it --rm --name fpm-test-2 -d php:8.0-fpm
docker exec -it fpm-test-2 bash
curl -I 127.0.0.1
curl: (7) Failed to connect to 127.0.0.1 port 80: Connection refused

I know port 80 is not used that is why I would expect to see connection refused not couldn't connect.
For me its just weird that both have different outputs and probably the cause of our issue.

[LaurentGoderre]

@mohammedx3 those 2 tags have different base debian version (Bookworm vs Bullseye) which coukld explain the difference in error messages, but regardless, curl is not a valid way to test FPM.

[bmerigan]

We have encountered an issue that sounds very similar if not the same.
Started happening 100% of the time after upgrading services running in Docker containers from PHP 8.1 to PHP 8.3.

We found a container which makes external requests (via a web-proxy) first will then fail when making internal requests to another container, with the error Host not found.
Our solution / workaround is to have 2 separate guzzle-http clients for internal and external calls.