autohttps: Implement auto_https prefer_wildcard option

[francislavoie]

Closes #5447

This implements a new auto_https prefer_wildcard option, which drops automation policies for non-wildcard domains when there's already a wildcard in another policy.

This allows users to flatten their config, and instead of using a pattern like https://caddyserver.com/docs/caddyfile/patterns#wildcard-certificates they can instead do something like:

{
	auto_https prefer_wildcard
}

*.example.com {
	tls {
		dns <provider>
	}
	respond "fallback"
}

foo.example.com {
	respond "foo"
}

This would only produce a single wildcard certificate, and no individual certificate for foo.example.com since it's already covered by the wildcard.

This also allows specifying multiple arguments to auto_https, so you can do the following to set multiple Automatic HTTPS options. Previously only one could be set at a time, which was generally fine because there wasn't actually any usecase where it would be useful:

{
	auto_https prefer_wildcard disable_redirects
}

I've only manually (visually) tested with a few simple usecases. Unfortunately we have a big lack of tests for the Automatic HTTPS logic because it manipulates config at runtime. I probably need help with testing this to make sure it doesn't have weird side effects. Thankfully, this should be safe/backwards compatible as long as users don't enable this option. We could call it experimental for now.

[mholt]

Thanks Francis, this looks appealing. Will wait for one or two people to field-test it.

[abjugard]

This looks amazing and a super useful feature to reduce subdomain discovery through certificate transparency!

I would suggest a small addition in order to minimise this even further.

E.g. to make Caddy always use wildcards.

This could maybe be implemented as another option for this directive, called force_wildcard, which figures out the minimal number of wildcard certs that need to be generated to cover all sites, and then uses your logic from this code to apply those certs to the correct sites.

Say I have two sites setup, for serviceA.domain.tld and serviceB.domain.tld, and use this setting, Caddy would then generate *.domain.tld and use it.

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

Alternatively, add another directive that allows users to specify a list of wildcard certs that will always be managed regardless of if they're used by sites or not, and then your feature here will be able to use those to avoid generating specific certs. A little less "magic" than the force_wildcard idea, but probably much easier to implement.

What do you think about this @francislavoie?

I guess it might be necessary to give Caddy a list of domains we control in order for this to work when using multiple domains so it doesn't see domain1.tld and domain2.tld and tries to generate *.tld... 😂

[JeDaYoshi]

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

@abjugard *.domain.tld won't cover sub-sub-domains. Caddy would need to generate certs for *.serviceA.domain.tld and *.serviceB.domain.tld. At least Let's Encrypt doesn't allow for *.*.domain.tld, AFAIK.

[abjugard]

If I have hostX.serviceA.domain.tld, hostY.serviceA.domain.tld, hostX.serviceB.domain.tld, hostY.serviceB.domain.tld, then all of those would be covered by the *.*.domain.tld certificate (does ACME even allow such certificates? 🤔).

@abjugard *.domain.tld won't cover sub-sub-domains. Caddy would need to generate certs for *.serviceA.domain.tld and *.serviceB.domain.tld. At least Let's Encrypt doesn't allow for *.*.domain.tld, AFAIK.

Right, unfortunate limitation but understandable.

Then I think we can simplify the behaviour and not need to specify which domains are owned, but just let Caddy automatically figure out to make wildcards for the leftmost label.

Perhaps we should just wait for this PR to get merged then I can take a crack at a force-wildcard PR?

[mholt]

I'm actually implementing part of this in CertMagic -- likely will become a "subject transformer" that allows changes to the subject name when managing certs, so, e.g. one can lob off the leftmost label and replace it with * to make it a wildcard. Of course, the Caddy interface will still need to be developed, which this PR could be. But I kind of want to come back to this after CertMagic has its change made, since it coincides with caddyserver/certmagic#280.

[mholt]

@francislavoie How would you feel if we/(I?) updated this PR to use the new SubjectTransformer introduced in the linked issue/commit? It should be relatively simple I think, if the global option is set, then set the SubjectTransformer for CertMagic configs to be a 3-line function.

[francislavoie]

I'm not sure how that would look 🤔 maybe make a branch off this one if you think it's simple?

[mholt]

I will try soon! :)

[mholt]

Actually the SubjectTransformer might be slightly tangential to this, rather than directly related -- the transformer is for, like, "I have a.b.c and b.b.c, but I want you to manage a single wildcard instead of individual certs for each specific domain." Whereas this change is, "I have *.b.c and a.b.c, and I want a.b.c to use the wildcard cert."

There's another aspect I want to consider as well, that is some users want just specific domains to be served under a wildcard, while the others shouldn't be. For example, in the config above, if there was also a site, bar.example.com, but they wanted that served with its own cert (maybe to try to obscure the fact that other subdomains are being served?) how would they do that?

I think I want to give this more thought, even before I implement anything here.

This is a needed change though and I think it's a good start. I just don't want to commit to its API/syntax/implementation quite yet until we have a better picture of the bigger picture.

[francislavoie]

There's another aspect I want to consider as well, that is some users want just specific domains to be served under a wildcard, while the others shouldn't be. For example, in the config above, if there was also a site, bar.example.com, but they wanted that served with its own cert (maybe to try to obscure the fact that other subdomains are being served?) how would they do that?

I think that's already handled by my approach, because prefer_wildcard only applies to a subdomain if there's already a wildcard cert in the config being managed (in which case it uses that policy). For other domains if you just don't have a wildcard that covers it, it'll still make a cert for that single domain. If you want to opt-out for just one domain that's covered by a wildcard, then don't use this feature and do it the handle way 🤷‍♂️

[mholt]

Ah, okay. Hmm.

I might still wait on this until after 2.8 so I can give this a little more thought. We now have a way, in CertMagic, of mapping/transforming one subject name (domain name) to another, for the purposes of cert management. Even if this PR ends up being good as-is, I just want a little more time on it.

[omltcat]

Glad there is a PR for this and I really look forward to it. Thank you for your great work. One big advantage of having separate site blocks is when using caddy-docker-proxy labels.

Since these labels can be distributed across multiple docker-compose files, there can be some possibilities of misconfiguration somewhere (especially done by multiple people). With the current handle approach, if one of the subdomain is misconfigured, causes the ENTIRE wildcard site block to be removed, bringing down everything. With this PR, the failure would at least be localized (hopefully😊)

[polarathene]

If you want to opt-out for just one domain that's covered by a wildcard, then don't use this feature and do it the handle way 🤷‍♂️

Could there not just be a tls or similar directive for a more explicit opt-out? (I don't need such functionality myself though)

It already seems to be an issue according to this report where an internal wildcard cert is being used instead of the LetsEncrypt one for an explicit site address.

Alternatively, you could go the other way around like with local_certs / tls internal, and instead have something like tls internal_wildcard or prefer_wildcard in the actual tls directive options? (assuming that could also be used to prefer FQDN as an override too).

---

Since the certmagic subject transformer feature is available and Caddy 2.8 is released, is there anything that can be done to assist moving this feature forward?

I've only manually (visually) tested with a few simple usecases. Unfortunately we have a big lack of tests for the Automatic HTTPS logic because it manipulates config at runtime.
I probably need help with testing this to make sure it doesn't have weird side effects.

If you can provide a rough outline of what to test, I could put together configs to verify?

---

One potential bug (without this PR) that already appears to exist is assigning a domain a wildcard cert from external files, and another site block for that domain with a different subdomain implicitly using the wildcard. (EDIT: Nope, that was due to incorrect auto_https mode set)

[mholt]

I am now wondering if we should make preferring wildcards the default behavior as @abjugard suggested above.

In Slack, it was expressed that it would be a breaking change, but I am not sure if there are (m)any(?) use cases that would actually break. A wildcard cert is just as good as a subdomain cert.

[polarathene]

EDIT: Ignore me. I mistook the "prefer" wildcard cert to prefer provisioning an explicit subdomain with a wildcard cert (even if no wildcard was explicitly configured/requested).

---

A wildcard cert is just as good as a subdomain cert.

Isn't it generally considered better practice to prefer explicit SAN for certs than a wildcard? Some businesses may have compliance requirements related to that expectation?

That concern is only relevant if the private key was compromised for the attacker to leverage it before expiry/detection, but that may be sufficient time for the attacker? 🤷‍♂️

I don't operate at a scale where I'm paranoid about such personally, but I could understand it being a compliance concern elsewhere, which would make it a breaking change for those users? Thus you may want to introduce as opt-in, and switch to opt-out when a breaking change is acceptable?

Unless you don't consider such implicit assumptions/trust in software to be a breaking change? You could also take the route of adding a notice (to release notes and pinned issue) for awareness before switching to opt-out, giving any potential users time to become aware of the change landing in future?

To be fair though, I think when compliance matters that upgrading to newer releases would have a more strict policy than trusting semantic versioning 😅 So perhaps I'm rambling about a non-issue.

[kanashimia]

In Slack, it was expressed that it would be a breaking change, but I am not sure if there are (m)any(?) use cases that would actually break. A wildcard cert is just as good as a subdomain cert.

If Caddy fails to obtain cert for *.example.com then foo.example.com won't have any cert too right? That seems problematic, especially because LetsEncrypt won't give you wildcard cert without DNS-01 challenge.
Anyways, default behavior can be changed later.

[abjugard]

If Caddy fails to obtain cert for *.example.com then foo.example.com won't have any cert too right? That seems problematic, especially because LetsEncrypt won't give you wildcard cert without DNS-01 challenge.

The scenario you describe here isn't really a concern as the requirements for procuring a wildcard cert are strictly different from certs for specific subdomains. Caddy (I'm guessing) won't try to get a wildcard cert unless its able to, and if its able to then it will either succeed or fail in which case you've configured it wrong and it should correctly refuse to continue starting up the site in question.

[mholt]

@kanashimia

If Caddy fails to obtain cert for *.example.com then foo.example.com won't have any cert too right?

Wildcard certs require DNS challenges which have to be explicitly configured. If someone is configuring a wildcard into their server, then they probably don't intend to get subdomain certs at all. IMO.

Anyways, default behavior can be changed later.

Well, if we default to always preferring the wildcard cert, that changes this PR. I just want to do what is most expected and intuitive, and I feel like setting up wildcard hostnames is juuust explicit and involved enough that a user probably wants to use that cert for the rest of the subdomains.

Maybe an inverse of this PR would be useful then, that is, to ignore wildcard certs.

[mholt]

This will go out with 2.9 beta 1. Would appreciate some field testing!

[coandco]

I think I'm missing something about this. When I use a single wildcard domain, everything works as expected and it gets a cert. If I try to define multiple wildcard domains, it fails with no solvers available for remaining challenges (configured=[http-01 tls- alpn-01] offered=[dns-01] remaining=[dns-01]) and when I look at the resulting caddy JSON from http://localhost:2019/config/ the DNS solver info didn't make it in. I would expect defining multiple wildcard domains to result in multiple wildcard certs which the other domains can use, not for all of them to fail. I've attached the Caddyfiles and resulting JSON configs to this comment.

multiple.Caddyfile.txt
multiple.json
single.Caddyfile.txt
single.json

[francislavoie]

Interesting, thanks @coandco, looks like the Caddyfile adapter part is consolidating automation policies too aggressively. I'll try to make a fix.

[francislavoie]

Fixed in #6636 @coandco, thanks for the test cases!

[rirze]

Adding to the earlier conversation about excluding some subdomains from using the wildcard domain; here's an example that applies to me:

I'd like to use Cloudflare's DNS proxy on the wildcard record, except for one subdomain, which I want to go directly to server's IP. It would be nice if Caddy could exclude that one subdomain from the catch-all.

[francislavoie]

Wait, are you saying you want one wildcard to override all subdomain, but another wildcard to not override subdomains? That's uh... wild. Sigh.

[rirze]

I might be misspeaking here-- I only want one wildcard to cover all subdomains defined. But I would like one specific subdomain to be excluded from the wildcard...

To be honest, thinking about it more, I think managing the records manually is easier since it's just one site. Sorry for the confusion.

[polarathene]

Presently the docs have no coverage of this feature?

Hope it's ok to document here in the meantime for those interested in it. I wasn't 100% sure of some behaviour, and someone recently asked me about this feature with the ability to exclude specific sites (seems to be a common expectation / requirement), while not that discoverable yet - the new tls force_automate feature resolves that (I'm aware it's currently considered experimental).

---

I only want one wildcard to cover all subdomains defined. But I would like one specific subdomain to be excluded from the wildcard

@rirze you would use auto_https prefer_wildcard with something configured to load/provision a wildcard certificate, then all other site-addresses that would be able to use that would prefer the wildcard instead of provisioning their own separate certificates.

To have a single address opt-out of the wildcard preference, you'd then use the other related feature tls force_automate that landed with the Caddy 2.9 release.

Here's an example Caddyfile, just replace .internal with your own TLD:

{
  auto_https prefer_wildcard
}

# Provision a wildcard for your domain (or load an external one via `tls` directive):
# NOTE: Order of site blocks declare is not important for the feature availability.
*.internal {
  #tls /srv/certs/example.internal/cert.pem /srv/certs/example.internal/key.pem
  abort
}

# Opt-out of preferring wildcard cert (or externally loaded cert):
not-wild.internal {
  tls force_automate
  respond "Individual SAN cert provisioned"
}

wild.example.internal {
  respond "This prefers to use the wildcard cert"
}

# Does not implicitly provision wildcards:
not-wild.localhost {
  respond "Caddy needs to match an existing wildcard to skip provisioning a new cert"
}

---

Reference config

Self-isolated compose.yaml example for anyone interested in a quick throwaway test setup. Just copy/paste to a local compose.yaml file and run:

# Run Caddy container:
$ docker compose up -d --force-recreate

# Query each address and report back the SANs from the certificate provided:
$ docker compose run --rm --quiet-pull verify-certs

╭───────────────────────────┬──────────────────────────────╮
│      site-addresses       │             sans             │
├───────────────────────────┼──────────────────────────────┤
│ wild.localhost            │ *.localhost                  │
│ wild.example.test         │ example.test, *.example.test │
│ also-wild.example.test    │ example.test, *.example.test │
│ not-wild.localhost        │ not-wild.localhost           │
│ not-wild.example.test     │ not-wild.example.test        │
│ not-wild.example.internal │ not-wild.example.internal    │
╰───────────────────────────┴──────────────────────────────╯

It's mostly verbose for supporting the verify-certs container to better illustrate the different variants configured in Caddy.

Additional commentary added below in the config regarding curl which is commonly used, will be unreliable for verifying a *.localhost wildcard certificate since their 7.78.0 (July 2021) release.

Quick overview:

• Containers + configs:
  • Caddy container + Caddyfile (embedded)
  • Alpine container + check-certs shell script (embedded)
• Wildcards + non-wildcard certs tested (as shown in above output):
  • *.example.test => External CA + Leaf provisioned certs (embedded at end of file)
  • *.localhost + not-wild.* => Caddy provisioned certs via local_certs / tls internal

networks:
  default:
    name: example-wildcard
    ipam:
      driver: default
      config:
        - subnet: "172.16.42.0/24"

volumes:
  caddy_data:
    name: example-caddy-data

services:
  reverse-proxy:
    image: caddy:2.9
    volumes:
      - caddy_data:/data
    configs:
      - source: caddy-config
        target: /etc/caddy/Caddyfile
      - source: tls-leaf-cert-ecdsa-example
        target: /tls/example.test/cert.pem
      - source: tls-leaf-key-ecdsa-example
        target: /tls/example.test/key.pem
    ports:
      - "80:80"
      - "443:443"
      - "443:443/udp" # HTTP/3
    networks:
      default:
        ipv4_address: 172.16.42.11
        # Docker's embedded DNS will resolve these addresses to
        # this Caddy container:
        aliases:
          - wild.example.test
          - also-wild.example.test
          - not-wild.example.test
          - not-wild.example.internal

  # Optional: This service is for conveniently verifying correct cert provisioning:
  verify-certs:
    init: true
    # Scope to a profile (avoids starting this service by default):
    profiles:
      - cli
    # Local build (no need to try pull `image` remotely):
    pull_policy: build
    image: localhost/verify-certs
    build:
      dockerfile_inline: |
        FROM alpine
        RUN apk add --no-cache step-cli jq bash csview ca-certificates
        ENTRYPOINT /usr/local/bin/check-certs
    # Add to `/etc/hosts` for resolving DNS correctly instead of 127.0.0.1:
    # NOTE: curl ignores any `/etc/hosts` entries for the `localhost` TLD.
    # Use `curl --connect-to ::172.16.42.11 https://not-wild.localhost` to connect
    # to the Caddy container successfully, however curl will also fail to verify
    # any cert for an FQDN that relies on a `*.localhost` wildcard SAN.
    extra_hosts:
      - "wild.localhost=172.16.42.11"
      - "not-wild.localhost=172.16.42.11"
    volumes:
      - caddy_data:/data:ro
    configs:
      - source: check-certs
        target: /usr/local/bin/check-certs
        # Make the file executable to run the script via it's shebang:
        mode: 755
      - source: tls-ca-cert-ecdsa
        target: /tls/example.test/ca-root.pem

# The `$$` used below opts-out of the variable interpolation feature from Docker Compose
configs:
  check-certs:
    content: |
      #! /usr/bin/env bash

      # Caddy's self-signed root CA (used for `local_certs`):
      # Required for successfully verifying chain of trust during the inspect query:
      # Effectively copies each cert into /usr/local/share/ca-certificates
      # and runs `update-ca-certificates` to install into the system trust store.
      step certificate install /data/caddy/pki/authorities/local/root.crt > /dev/null
      step certificate install /tls/example.test/ca-root.pem > /dev/null

      SITES=(
        "wild.localhost"
        "wild.example.test"
        "also-wild.example.test"
        "not-wild.localhost"
        "not-wild.example.test"
        "not-wild.example.internal"
      )

      # Store results in TSV for better display formatting:
      echo -e 'site-addresses\tsans' > /tmp/results.tsv

      for FQDN in "$${SITES[@]}"; do
        # Parse JSON response for SANs and format into a string:
        DNS_NAMES=$(
          step certificate inspect --format json "https://$${FQDN}" \
          | jq -c '.extensions.subject_alt_name.dns_names | join(", ")'
        )

        echo -e "$${FQDN}\t$${DNS_NAMES}" >> /tmp/results.tsv
      done

      # Render a table from the TSV formatted data:
      csview --tsv --style rounded --disable-pager /tmp/results.tsv

  caddy-config:
    content: |
      # Global Settings
      {
        # Optional: For testing purposes (otherwise defaults to a public CA)
        # Have Caddy provision certs locally (self-signed):
        local_certs

        # If Caddy is already aware of a wildcard certificate for the SAN / FQDN,
        # have it prefer to use that instead of provisioning a separate cert:
        auto_https prefer_wildcard
      }

      :1337 {
        respond "Hello World!"
      }

      # This site-block will trigger automatic cert provisioning / renewal
      # for this wildcard domain (only required if configuring via Caddyfile):
      # NOTE: More specific site-addresses have priority when routing a request to
      # a site block, while this wildcard site-block acts as a fallback.
      *.localhost {
        # Reject fallback traffic received (subdomain did not match elsewhere)
        abort
      }

      # Force a certificate to be automatically provisioned regardless of any
      # existing valid provisioned certificates found (eg: a compatible wildcard)
      not-wild.localhost, not-wild.example.test {
        tls force_automate
        respond "Individual SAN certs provisioned"
      }

      # Although this is a specific site-address, the external cert loaded provides
      # a wildcard SAN which Caddy will recognize for wildcard preference elsewhere
      wild.example.test {
        tls /tls/example.test/cert.pem /tls/example.test/key.pem
        reverse_proxy :1337
      }

      # The first two will use the wildcard certs Caddy has provisioned or loaded,
      # The last site-address is provisioned with a new certificate (non-wildcard),
      # as no existing certificate has a valid SAN entry match for the FQDN:
      wild.localhost, also-wild.example.test, not-wild.example.internal {
        reverse_proxy :1337
      }

  # Optional - Remaining files are an externally provisioned cert example
  # Provisioned for `*.example.test` + `example.test` (via `step certificate create`):
  tls-leaf-cert-ecdsa-example:
    content: |
      -----BEGIN CERTIFICATE-----
      MIIB0DCCAXagAwIBAgIRANQ/OtiKq5l49rHqweg0tQkwCgYIKoZIzj0EAwIwHDEa
      MBgGA1UEAxMRU21hbGxzdGVwIFJvb3QgQ0EwHhcNMjQwMTAxMDAwMDAwWhcNMzQw
      MTAxMDAwMDAwWjAZMRcwFQYDVQQDEw5TbWFsbHN0ZXAgTGVhZjBZMBMGByqGSM49
      AgEGCCqGSM49AwEHA0IABLgdMSPKbiATi8GVm0rdAtZeBN+XxCBcYkf696fIPEfc
      Z3LuU2P1ikmPx7f+UhGqSBK5e7CewfZatoA0F+lzl9WjgZswgZgwDgYDVR0PAQH/
      BAQDAgeAMB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAdBgNVHQ4EFgQU
      FzcbZ2PsKlBSNv8EVcrSmVWPKdwwHwYDVR0jBBgwFoAUpaV6oR6VlDl+0RXY3qXK
      I5SizQ8wJwYDVR0RBCAwHoIMZXhhbXBsZS50ZXN0gg4qLmV4YW1wbGUudGVzdDAK
      BggqhkjOPQQDAgNIADBFAiBJvyURWSG8gJ2/ykcZ5ipudXe7/D0COd4EP3ks4VxN
      aQIhALjQoeJzwLHTqnbOENedTEVdOI/j/0Wr38JhJht0lgdy
      -----END CERTIFICATE-----

  tls-leaf-key-ecdsa-example:
    content: |
      -----BEGIN EC PRIVATE KEY-----
      MHcCAQEEIClhaEYEdVTKSqG7mbMbdKE/uB1wd1ncEcwwe1kULbR+oAoGCCqGSM49
      AwEHoUQDQgAEuB0xI8puIBOLwZWbSt0C1l4E35fEIFxiR/r3p8g8R9xncu5TY/WK
      SY/Ht/5SEapIErl7sJ7B9lq2gDQX6XOX1Q==
      -----END EC PRIVATE KEY-----

  tls-ca-cert-ecdsa:
    content: |
      -----BEGIN CERTIFICATE-----
      MIIBfTCCASKgAwIBAgIRANvAFk7fgnv6LBBO8KMJ7BswCgYIKoZIzj0EAwIwHDEa
      MBgGA1UEAxMRU21hbGxzdGVwIFJvb3QgQ0EwHhcNMjEwMTAxMDAwMDAwWhcNMzEw
      MTAxMDAwMDAwWjAcMRowGAYDVQQDExFTbWFsbHN0ZXAgUm9vdCBDQTBZMBMGByqG
      SM49AgEGCCqGSM49AwEHA0IABDaUnm6W6pQTSVcP+rl+QqwwpzrLuDVMXHGvQtun
      1si87g04a1462KQAZCUtmnAIJiEowvcXMQaS2dm8Z0qf28GjRTBDMA4GA1UdDwEB
      /wQEAwIBBjASBgNVHRMBAf8ECDAGAQH/AgEBMB0GA1UdDgQWBBSlpXqhHpWUOX7R
      FdjepcojlKLNDzAKBggqhkjOPQQDAgNJADBGAiEA/R5frrFFWY3441N1CncFaMlY
      9b/rUJKU6rh0eezXKHwCIQCTZ4D2/HPX06betNj8d/bTG16iAqoqgLUc+6PKc3PV
      sA==
      -----END CERTIFICATE-----

  # This CA key is only needed if provisioning more leafs / intermediates:
  tls-ca-key-ecdsa:
    content: |
      -----BEGIN EC PRIVATE KEY-----
      MHcCAQEEIFfxEAiTHCjehLSCydAt6v4Qt1MCp0OpW7QIrQuZ7JMeoAoGCCqGSM49
      AwEHoUQDQgAENpSebpbqlBNJVw/6uX5CrDCnOsu4NUxcca9C26fWyLzuDThrXjrY
      pABkJS2acAgmISjC9xcxBpLZ2bxnSp/bwQ==
      -----END EC PRIVATE KEY-----

Reproduction for the curl concern for reference (verified with curl in a glibc based Fedora 41 container too FWIW):

# Shell into the Alpine container:
$ docker compose run --rm -it --entrypoint /bin/ash verify-certs

# Add both private CA certs into the trust store, then install curl:
$ check-certs && apk add curl

# Explicit FQDNs in a SAN work fine:
$ curl --connect-to ::172.16.42.11 https://not-wild.localhost
Hello World!

# Despite the certificate chain of trust being valid, curl fails to validate this wildcard SAN:
$ curl --connect-to ::172.16.42.11 https://wild.localhost
curl: (60) SSL: no alternative certificate subject name matches target hostname 'wild.localhost'
More details here: https://curl.se/docs/sslcerts.html

# Yet this does not happen for a wildcard for another TLD like `example.test`:
# It would fail if the CA root was not installed into the trust store prior.
$ curl --connect-to ::172.16.42.11 https://wild.example.test
Hello World!

---

UPDATE: The above reference example only checks the SANs by DNS names, if your certificate relies on the CN still or other means those would not have been listed in the table.

Additionally if Caddy has a certificate loaded with an exact match of the site-address, it seems it'll prefer that even when it's externally loaded, which might not be how tls force_automate was intended to work? (see this issue for more info)

[rirze]

Interesting, I will give this a try! Thanks for the detailed explanation.

[drglove]

Does this feature work with the bind directive? e.g.

{
	auto_https prefer_wildcard
}

*.example.com {
	tls {
		dns <provider>
	}
	bind 10.0.0.100
	respond "fallback"
}

foo.example.com {
        bind 10.0.0.101
	respond "foo"
}

[francislavoie]

Try it. Looks at the output of caddy adapt -p and see what you get.

[drglove]

Try it. Looks at the output of caddy adapt -p and see what you get.

Thanks, I wasn't aware of that command to see more detailed output. It appears to work. The reason I ask is because it wasn't possible (to my knowledge) to use wildcards in conjunction with bind since that must be specified at too high a level (in my case *.testdomain.com with handle statements). The fact that I can do this with this change is excellent, much appreciated. I will check that it behaves as I expect but the config looks good.

For completeness, here is my test config and the resulting JSON configuration printed.
Caddyfile:

{
        auto_https prefer_wildcard
}

*.testdomain.com {
        bind 10.0.50.125
        abort
}

foo.testdomain.com {
        bind 10.0.90.104
        abort
}

results in this correct config from caddy adapt:

{
        "apps": {
                "http": {
                        "servers": {
                                "srv0": {
                                        "listen": [
                                                "10.0.50.125:443"
                                        ],
                                        "routes": [
                                                {
                                                        "match": [
                                                                {
                                                                        "host": [
                                                                                "*.testdomain.com"
                                                                        ]
                                                                }
                                                        ],
                                                        "handle": [
                                                                {
                                                                        "handler": "subroute",
                                                                        "routes": [
                                                                                {
                                                                                        "handle": [
                                                                                                {
                                                                                                        "abort": true,
                                                                                                        "handler": "static_response"
                                                                                                }
                                                                                        ]
                                                                                }
                                                                        ]
                                                                }
                                                        ],
                                                        "terminal": true
                                                }
                                        ],
                                        "automatic_https": {
                                                "prefer_wildcard": true
                                        }
                                },
                                "srv1": {
                                        "listen": [
                                                "10.0.90.104:443"
                                        ],
                                        "routes": [
                                                {
                                                        "match": [
                                                                {
                                                                        "host": [
                                                                                "foo.testdomain.com"
                                                                        ]
                                                                }
                                                        ],
                                                        "handle": [
                                                                {
                                                                        "handler": "subroute",
                                                                        "routes": [
                                                                                {
                                                                                        "handle": [
                                                                                                {
                                                                                                        "abort": true,
                                                                                                        "handler": "static_response"
                                                                                                }
                                                                                        ]
                                                                                }
                                                                        ]
                                                                }
                                                        ],
                                                        "terminal": true
                                                }
                                        ],
                                        "automatic_https": {
                                                "prefer_wildcard": true
                                        }
                                }
                        }
                }
        }
}

[drglove]

Upon further inspection, it doesn't appear to play well enough with bind directives. Take this sample Caddyfile:

{
        auto_https prefer_wildcard
}

*.testdomain.com {
        bind 10.0.50.125
        abort
}

foo.testdomain.com {
        bind 10.0.50.125
        abort
}

bar.testdomain.com {
        bind 10.0.90.104
        abort
}

Note that *.testdomain.com and foo.testdomain.com share the same bind address, but bar.testdomain.com does not.

caddy adapt gives the following:

{
	"apps": {
		"http": {
			"servers": {
				"srv0": {
					"listen": [
						"10.0.50.125:443"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"foo.testdomain.com"
									]
								}
							],
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"abort": true,
													"handler": "static_response"
												}
											]
										}
									]
								}
							],
							"terminal": true
						},
						{
							"match": [
								{
									"host": [
										"*.testdomain.com"
									]
								}
							],
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"abort": true,
													"handler": "static_response"
												}
											]
										}
									]
								}
							],
							"terminal": true
						}
					],
					"automatic_https": {
						"skip_certificates": [
							"foo.testdomain.com"
						],
						"prefer_wildcard": true
					}
				},
				"srv1": {
					"listen": [
						"10.0.90.104:443"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"bar.testdomain.com"
									]
								}
							],
							"handle": [
								{
									"handler": "subroute",
									"routes": [
										{
											"handle": [
												{
													"abort": true,
													"handler": "static_response"
												}
											]
										}
									]
								}
							],
							"terminal": true
						}
					],
					"automatic_https": {
						"prefer_wildcard": true
					}
				}
			}
		}
	}
}

Note that the skip_certificates entry only appears for those under the same server with the same bind directive

"skip_certificates": [
	"foo.testdomain.com"
]

As a result, I've observed that even with auto_https prefer_wildcard being set, a certificate is being requested unexpectedly for bar.testdomain.com (or the equivalent for a domain I actually control). In my case, this means details of internal infrastructure is known to the outside world, where it would have been preferable to keep it hidden.