Implement issue #6296 passing FDs / socket activation

[MayCXC]

third time's the charm ☘️

example Caddyfile:

{
	auto_https disable_redirects
	admin off
}

http://localhost {
	bind fd/{env.CADDY_HTTP_FD} {
		protocols h1
	}
	log
	respond "Hello, HTTP!"
}

https://localhost {
	bind fd/{env.CADDY_HTTPS_FD} {
		protocols h1 h2
	}
	bind fdgram/{env.CADDY_HTTP3_FD} {
		protocols h3
	}
	log
	respond "Hello, HTTPS!"
}

as seen above, two reserved networks fd and fdgram have been added. fd/3 creates a FileListener from fd 3, fdgram/4 creates a FilePacketConn from fd 4. this simplifies configuration compared to #6543 , but the "actual" host addresses are no longer visible. so, for health checks and the admin interface, fds are treated similarly to unix sockets.

this is a good thing i think, fds should be decoupled from the address they are bound to. separate config options can be added to override the origin address for health checks / the admin interface if needed.

link to issue: #6296

[wplinge]

I'm really looking forward to this one and have been trying it out.

I listen for multiple hosts so was trying to use default_bind instead. I'd normally assume I was holding it wrong and/or file another bug after this landed, but it led to a panic around HTTP/3 handling rather than a normal error so might be a more fundamental type problem.

With Caddyfile

{
	auto_https disable_redirects
	admin off
	default_bind fd/3 {
		protocols h1
	}
	default_bind fd/4 {
		protocols h1 h2
	}
	default_bind fdgram/5 {
		protocols h3
	}
}
https://localhost {
	log
	respond "Wibble"
}

I get the backtrace

Sep 26 12:21:17 sandbox caddy[131974]: panic: interface conversion: caddy.deletePacketConn is not net.Listener: missing method Accept
Sep 26 12:21:17 sandbox caddy[131974]:
Sep 26 12:21:17 sandbox caddy[131974]: goroutine 1 [running]:
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Start(0xc000bce300)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/app.go:533 +0x14f4
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.run.func1({{0x2051c50, 0xc00088d9a0}, 0xc000b9f890, 0xc000e80180, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:420 +0x111
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.run(0x1000100?, 0x1)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:436 +0x198
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.unsyncedDecodeAndRun({0xc0001029a0, 0x156, 0x160}, 0x1)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:343 +0x145
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.changeConfig({0x1af13ed, 0x4}, {0x1af8455, 0x7}, {0xc000102840, 0x156, 0x160}, {0x0, 0x0}, 0x1)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:234 +0x6b6
[... other functions that look like generic startup stuff ...]

Let me know if you'd prefer a separate issue (or different channel entirely) for this.

[MayCXC]

I'm really looking forward to this one and have been trying it out.

I listen for multiple hosts so was trying to use default_bind instead. I'd normally assume I was holding it wrong and/or file another bug after this landed, but it led to a panic around HTTP/3 handling rather than a normal error so might be a more fundamental type problem.

With Caddyfile

{
	auto_https disable_redirects
	admin off
	default_bind fd/3 {
		protocols h1
	}
	default_bind fd/4 {
		protocols h1 h2
	}
	default_bind fdgram/5 {
		protocols h3
	}
}
https://localhost {
	log
	respond "Wibble"
}

I get the backtrace

Sep 26 12:21:17 sandbox caddy[131974]: panic: interface conversion: caddy.deletePacketConn is not net.Listener: missing method Accept
Sep 26 12:21:17 sandbox caddy[131974]:
Sep 26 12:21:17 sandbox caddy[131974]: goroutine 1 [running]:
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2/modules/caddyhttp.(*App).Start(0xc000bce300)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/modules/caddyhttp/app.go:533 +0x14f4
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.run.func1({{0x2051c50, 0xc00088d9a0}, 0xc000b9f890, 0xc000e80180, {0x0, 0x0, 0x0}, {0x0, 0x0, 0x0}, ...})
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:420 +0x111
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.run(0x1000100?, 0x1)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:436 +0x198
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.unsyncedDecodeAndRun({0xc0001029a0, 0x156, 0x160}, 0x1)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:343 +0x145
Sep 26 12:21:17 sandbox caddy[131974]: github.com/caddyserver/caddy/v2.changeConfig({0x1af13ed, 0x4}, {0x1af8455, 0x7}, {0xc000102840, 0x156, 0x160}, {0x0, 0x0}, 0x1)
Sep 26 12:21:17 sandbox caddy[131974]:         github.com/caddyserver/caddy/v2@v2.8.4/caddy.go:234 +0x6b6
[... other functions that look like generic startup stuff ...]

Let me know if you'd prefer a separate issue (or different channel entirely) for this.

that for sure looks like an issue with this branch. thank you for testing it, I can take a look later today.

[MayCXC]

@wplinge HEAD treats default_bind like bind now, so you can add multiple options with protocol blocks like in your example. if you can try running it again, it should create the expected listeners.

@mholt this was an additional feature, but it is compatible with every existing Caddyfile and config JSON. I also added a clearer error message for the case in the log.

[wplinge]

@MayCXC Thanks, it works perfectly for me with that update. I'd noticed only one default_bind was being used after posting, but still slightly surprised it led to that failure mode.

[MayCXC]

@MayCXC Thanks, it works perfectly for me with that update. I'd noticed only one default_bind was being used after posting, but still slightly surprised it led to that failure mode.

great, yeah what happened was only that last default_bind would end up in the config, and the protocols sub option was ignored for it. so instead of working like bind, the default h1/h2 protos were served with your datagram fd, but it definitely should have worked the way you expected. b98bf1a adds a clearer error message for the failure mode you saw, similar to the one you see when you try to serve h3 with a stream fd.

[mholt]

Well, impressive.

This change is too big for me to fully understand and thoroughly review. I appreciate the field testing, and think we could benefit from more of it. So let's go ahead and merge this, if you're ready, so it goes in Caddy 2.9 beta releases.

[MayCXC]

thanks, and yes I'm ready now. I will be one of the beta testers with production traffic :)

[mholt]

🤞

[francislavoie]

Unfortunately I think this broke integration tests. The annoying thing is the integration tests are horribly flaky, but with this commit on master I get 100% fail rate on a dozen retries instead of a pass in two tries on the prior commit in master.

Specifically, this is the failing test:

--- FAIL: TestHTTPRedirectWrapperWithLargeUpload (0.01s)
    listener_test.go:74: failed to post: Post "https://localhost:9443": write tcp 127.0.0.1:33560->127.0.0.1:9443: write: connection reset by peer
    caddytest.go:134: ----------- failed with config -----------
        {
          "admin": {
            "listen": "localhost:2999"
          },
          "apps": {
            "http": {
              "http_port": 9080,
              "https_port": 9443,
              "servers": {
                "srv0": {
                  "listen": [
                    ":9443"
                  ],
                  "listener_wrappers": [
                    {
                      "wrapper": "http_redirect"
                    },
                    {
                      "wrapper": "tls"
                    }
                  ],
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "127.0.0.1:42089"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "host": [
                            "localhost"
                          ]
                        }
                      ],
                      "terminal": true
                    }
                  ]
                }
              }
            },
            "pki": {
              "certificate_authorities": {
                "local": {
                  "install_trust": false
                }
              }
            },
            "tls": {
              "automation": {
                "policies": [
                  {
                    "issuers": [
                      {
                        "module": "internal"
                      }
                    ]
                  }
                ]
              }
            }
          }
        }

[MayCXC]

Unfortunately I think this broke integration tests. The annoying thing is the integration tests are horribly flaky, but with this commit on master I get 100% fail rate on a dozen retries instead of a pass in two tries on the prior commit in master.

Specifically, this is the failing test:

[...]

investigating this, it looks like I broke http_redirect. in this Caddyfile based off of func setupListenerWrapperTest in listener_test.go, commenting it out lets my browser connect to https://localhost:9443, but leaving it uncommented shows me a ERR_HTTP2_PROTOCOL_ERROR :

{
	skip_install_trust
	admin localhost:2999
	http_port     9080
	https_port    9443
	local_certs
	servers :9443 {
		listener_wrappers {
#			http_redirect
#			tls
		}
	}
}

localhost {
	log
	respond "tlspls"
}

so next I will debug http_redirect to see what is going wrong.

[MayCXC]

the above config with http_redirect uncommented works with just app.go reset to https://github.com/caddyserver/caddy/blob/1a345b4fa620dfb0909a3b086bd76e35dfdbefa5/modules/caddyhttp/app.go , so the bug is in very likely in func Start . bisecting from here should be straightforward.

[MayCXC]

fix is here: #6599