Caddy 2.9.1 -> 2.10.0 strange TLS issue

r00tsh3ll · June 21, 2025, 9:59am

1. The problem I’m having:

After an upgrade from 2.9.1 from 2.10.0 and the same Caddyfile, TLS errors are returned for specifc (not on-demand) hostnames (hpb.yolo.asia and web.cos.eu in Caddyfile) .

I found that adding tls force_automate to these two blocks resolved the issue and is back running as it was with v2.9.1.
Still, I wonder if this is the expected behavior in v2.10.0 or if it is a bug

2. Error messages and/or full log output:

{"level":"debug","ts":1750475797.387161,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"hpb.yolo.asia"}
{"level":"debug","ts":1750475797.387172,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.yolo.asia"}
{"level":"debug","ts":1750475797.3871813,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.asia"}
{"level":"debug","ts":1750475797.3871906,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*"}
{"level":"debug","ts":1750475797.3872213,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"1.46.12.74","remote_port":"64295","server_name":"hpb.yolo.asia","remote":"1.46.12.74:64295","identifier":"hpb.yolo.asia","cipher_suites":[4865,4866,4867,49195,49196,52393,49199,49200,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.0003,"load_or_obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1750475797.3872805,"logger":"http.stdlib","msg":"http: TLS handshake error from 1.46.12.74:64295: no certificate available for 'hpb.yolo.asia'"}

3. Caddy version:

v2.10.0 h1:fonubSaQKF1YANl8TXqGcn4IbIRUDdfAkpcsfI/vX5U=

4. How I installed and ran Caddy:

Installed via apt get (cloudsmith repo)

a. System environment:

Debian 12.11

b. Command:

systemctl start caddy

c. Service/unit/compose file:

The one provided by the package

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        admin 127.0.0.1:8888
        default_bind 127.0.0.1 [::1] 10.99.99.21 192.168.1.21
        grace_period 3s
        log {
                output file /var/log/caddy/caddy.log {
                        roll_size 250MiB
                        roll_keep_for 15d
                }
                level ERROR
        }
        email contact@cos.eu
        default_sni web.cos.eu
        servers {
                protocols h1 h2
        }
        on_demand_tls {
                ask http://127.0.0.80/ask.php
        }
}

(common) {
        # Return client IP
        @_client_ip {
                path /_client_ip
        }
        respond @_client_ip "{client_ip}" 200 {
                close
        }
        reverse_proxy http://127.0.0.80:80
}
(hsts) {
        header Strict-Transport-Security max-age=15752000;
}

# Specific entry for main host
web.cos.eu {
        import common
}

# Talk HPB for my.yolo.asia
hpb.yolo.asia {
        route /standalone-signaling/* {
                uri strip_prefix /standalone-signaling
                reverse_proxy http://127.0.0.1:8181
        }
        import hsts
}

# Catch all routes for the rest of the domains
http:// {
        import common
}

# Match only host names and not ip-addresses:
https://*.*,
https://*.*.*,
https://*.*.*.*,
https://*.*.*.*.*,
https://*.*.*.*.*.* {
        tls {
                on_demand
        }
        import hsts
        import common
}

Thank you for your feedback!

r00tsh3ll · June 21, 2025, 4:53pm

I tried to find some info about this, the closest topic I can find that might be related is:

Mohammed90 · June 21, 2025, 9:58pm

It’s a side-effect of this change

github.com/caddyserver/caddy

caddytls: Prefer managed wildcard certs over individual subdomain certs

master ← prefer-wildcard-default

opened 11:12PM - 16 Apr 25 UTC

mholt

+172 -491

I am refactoring `caddytls.TLS.Manage()` to take a `map[string]struct{}` for the… list of domains to manage certs for, instead of a `[]string` -- as part of a rethinking of how `prefer_wildcard` (from #6146) works. I've been giving this lots of thought, though I probably still have not thought of everything, but it seems to me that `prefer_wildcard` actually makes more sense at the TLS-app-level, rather than as part of automatic HTTPS in the HTTP app. See, my reasoning is that many users who want `prefer_wildcard` -- which I assume is most, but without telemetry, yet again, there's no way to be certain -- want it for two main reasons: 1) fewer certificates / rate limit avoidance, and 2) privacy. The first is a matter of availability, which is crucial, and the second is newly possible with ECH + wildcard certs. We want to try to be private-by-default whenever it is practical & feasible. Caddy manages certs for more than just HTTPS for sites, so we actually need to bring that logic into the TLS app itself so that we can properly deliver on those two goals: avoiding rate limits and offering privacy. If we only do it for auto-HTTPS, it's not as complete (or correct) of a solution. So, now, instead of `prefer_wildcard` -- which was definitely the right idea -- we'll make wildcard coverage the default, then have something like an `individual_certificates` bool in automation policies that allows users to opt OUT of this coverage, like for subdomains they want to have their own certificates. And the map makes existence queries for wildcards way faster than iterating a large slice. This may be surprising for some in that Caddy will no longer get certs for individual subdomains when used as site addresses in the Caddyfile for which there is a wildcard domain specified, but this seems correct most of the time. This behavior can be overridden by setting `individual_certificates: true` in the appropriate automation policy, but I haven't yet exposed this to the Caddyfile. @francislavoie -- any thoughts on how we should do this? It could be a global option, i.e. "any subdomains should just always have their own certificates" (which restores the previous behavior), OR we could make it per-site, so only certain subdomains might have their own certificates. (I know this PR technically rips out the fine code written by @francislavoie -- but I view it as building upon it and finishing it.) --- Before (and before prefer_wildcard), you'd need this Caddyfile structure (as documented in our Caddyfile Patterns page) in order to have multiple sites use the same wildcard certificate: ```caddy *.example.com { tls { dns <provider_name> [<params...>] } @foo host foo.example.com handle @foo { respond "Foo!" } @bar host bar.example.com handle @bar { respond "Bar!" } # Fallback for otherwise unhandled domains handle { abort } } ``` (I actually liked this structure because it was very clear to me what the cert management was doing.) Now, you can write it like this: ```caddy *.example.com { tls { dns <provider_name> [<params...>] } abort } foo.example.com { respond "Foo!" } bar.example.com { respond "Bar!" } ``` Kind of weird to have a route-less site, just for the sake of a wildcard, but whatever. Is this better/cleaner/easier? I dunno. But from a privacy perspective, I think this PR makes sense. (I guess technically it really needs ECH to be utilized in order for it to make sense. I could make it conditional on ECH being configured? Gets complicated though...)

It was mentioned in the release notes

r00tsh3ll · June 22, 2025, 8:25am

Oh okay I see (almost).

I was surprised that it did not request a certificate for these hosts let it be via the specific blocks for these hostnames or the on demande catch all, without having to add force_automate.