nixos/caddy: add support for caddy v2 (take 2)

[sephii]

Motivation for this change

The caddy module allows you to use Caddy 2 but it's broken (#92992). This PR is a new take at #86686, which has been dormant for a few weeks now. Since I'd really like to get Caddy 2 support in 20.09 I'm opening a new PR with the requested changes.

The tests pass when running the tests interactively (nix-build caddy.nix -A driver and then start_all() followed by test_script()), but they fail when running them non-interactively (nix-build caddy.nix) because the test directory created in the test cannot be found. Since I can't figure out why, any help is appreciated. (fixed!)

I would also like to bump the version to 2.1.1 instead of 2.0, but doing it gives me some go errors I don't know how to fix (eg. gopkg.in/yaml.v2@v2.3.0: is explicitly required in go.mod, but not marked as explicit in vendor/modules.txt), so if someone knows how to make it work, it would be nice. (fixed!)

Things done

• Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
• Built on platform(s)
  • NixOS
  • macOS
  • other Linux distributions
• Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
• Tested compilation of all pkgs that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review wip"
• Tested execution of all binary files (usually in ./result/bin/)
• Determined the impact on package closure size (by running nix path-info -S before and after)
• Ensured that relevant documentation is up to date
• Fits CONTRIBUTING.md.

[Br1ght0ne]

Thanks @sephii, I fully support your effort. I'll take a look at the tests.

[Br1ght0ne]

Besides failing tests, LGTM

Forgot about failing tests

[KamilaBorowska]

@ofborg test caddy

[KamilaBorowska]

From what I can tell, the issue is that Nix doesn't recognize paths in a configuration file as being inputs. caddy-config.json looks like this:

{
    "apps": {
        "http": {
            "servers": {
                "srv0": {
                    "automatic_https": {
                        "skip": [
                            "localhost"
                        ]
                    },
                    "listen": [
                        ":80"
                    ],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "handle": [
                                                {
                                                    "handler": "vars",
                                                    "root": "/nix/store/496k1zx0li2mp94xnydn2smk1gbybw2n-testdir"
                                                },
                                                {
                                                    "encodings": {
                                                        "gzip": {}
                                                    },
                                                    "handler": "encode"
                                                },
                                                {
                                                    "handler": "file_server",
                                                    "hide": [
                                                        "/nix/store/69yh559dagwrhfa8bahpm9v99sd2qkg8-Caddyfile"
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "localhost"
                                    ]
                                }
                            ],
                            "terminal": true
                        }
                    ]
                }
            }
        },
        "tls": {
            "automation": {
                "policies": [
                    {
                        "issuer": {
                            "ca": "https://acme-v02.api.letsencrypt.org/directory",
                            "email": "",
                            "module": "acme"
                        }
                    }
                ]
            }
        }
    }
}

Neither /nix/store/496k1zx0li2mp94xnydn2smk1gbybw2n-testdir nor /nix/store/69yh559dagwrhfa8bahpm9v99sd2qkg8-Caddyfile actually exist on a virtual machine.

This likely occurs due to importJSON here. Merging JSON files should likely be done as a derivation (probably using jq), not from Nix language.

  adaptedConfig = importJSON (pkgs.runCommand "caddy-config-adapted.json" { } ''
    ${cfg.package}/bin/caddy adapt \
      --config ${configFile} --adapter ${cfg.adapter} > $out
  '');
  configJSON = pkgs.writeText "caddy-config.json" (builtins.toJSON
    (recursiveUpdate adaptedConfig tlsConfig));

[KamilaBorowska]

Avoid use of importJSON here and merge JSON files in a derivation. The following should work.

  adaptedConfig = pkgs.runCommand "caddy-config-adapted.json" { } ''
    ${cfg.package}/bin/caddy adapt \
      --config ${configFile} --adapter ${cfg.adapter} > $out
  '';
  tlsJSON = pkgs.writeText "tls.json" (builtins.toJSON tlsConfig);
  configJSON = pkgs.runCommand "caddy-config.json" { } ''
    ${pkgs.jq}/bin/jq -s '.[0] * .[1]' ${adaptedConfig} ${tlsJSON} > $out
  '';

[sephii]

The tests should now pass, thanks for the help!

[KamilaBorowska]

@ofborg test caddy

[lf-]

I don't think this is a regression caused by this PR, but for some reason, it's emitting a StartLimitIntervalSec in the wrong section (should be under [Unit] as of some time in 2017 according to the mailing list).

Here's the generated service file:

[Unit]
After=network-online.target
Description=Caddy web server
Wants=network-online.target

[Service]
# ... snip ...
StartLimitIntervalSec=14400
TimeoutStopSec=5s
Type=simple
User=caddy

[sephii]

You're right, it should be moved to the Unit section along with StartLimitBurst. I think you should file an issue, since this problem is present in other services (grepping for StartLimitIntervalSec and StartLimitBurst reveals that other modules are using these options, and that they're also in the Service section).

[lf-]

@sephii I should file a PR and fix all of them as it's relatively trivial. ~~Not sure when I'll have time to do that though.~~Well, the following bug is a blocker on my work, so I guess I have time now!

[lf-]

In terms of actual bugs in this PR though:

@sephii

There's a bug introduced in this PR where we are doing a bit of hamfisted overwriting of the tls key in our generated configs, breaking tls internal settings in the config section. Here's a repro:

{pkgs, ...}: {
    services.caddy.package = pkgs.caddy;
    services.caddy.enable = true;
    services.caddy.agree = true;
    services.caddy.email = "letsencrypt@catgirl.enterprises";
    services.caddy.ca = "https://acme-staging-v02.api.letsencrypt.org/directory";
    services.caddy.config = ''
      test.catgirl.enterprises:443 {
        encode zstd gzip
        log
        tls internal
        reverse_proxy localhost:3000
      }
    '';
}

is making the following bad config:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "test.catgirl.enterprises"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "encodings": {
                            "gzip": {},
                            "zstd": {}
                          },
                          "handler": "encode"
                        },
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "localhost:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "logs": {}
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "issuer": {
              "ca": "https://acme-staging-v02.api.letsencrypt.org/directory",
              "email": "letsencrypt@catgirl.enterprises",
              "module": "acme"
            }
          }
        ]
      }
    }
  }
}

Compare the tls section to what caddy adapt produces for the equivalent caddyfile with the content of config:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "test.catgirl.enterprises"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "encodings": {
                            "gzip": {},
                            "zstd": {}
                          },
                          "handler": "encode"
                        },
                        {
                          "handler": "reverse_proxy",
                          "upstreams": [
                            {
                              "dial": "localhost:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ],
          "logs": {}
        }
      }
    },
    "tls": {
      "automation": {
        "policies": [
          {
            "subjects": [
              "test.catgirl.enterprises"
            ],
            "issuer": {
              "module": "internal"
            }
          }
        ]
      }
    }
  }
}

[lf-]

Update: it looks like it's that for some reason jq is not merging arrays while doing a * recursive merge. I don't know enough jq to fix it though, sorry. I'll instead go fix the systemd unit option issues [EDIT: done this fix, see #97512] :)

[sephii]

I think the following jq should do the trick: jq -s '.[0] * {"apps": {"tls": {"automation": {"policies": (if .[0].apps.tls.automation.policies == .[1]?.apps.tls.automation.policies then .[0].apps.tls.automation.policies else (.[0].apps.tls.automation.policies + .[1]?.apps.tls.automation.policies) end)}}}}'. I don't know jq either so there might be a better way to do it. Can you please try and let me know, so I can submit a patch?