On Demand TLS with custom CA and nested Wildcards

1. The problem I’m having:

My organisation has automatic DNS records that point to developer machines so they can use real devices to access their local development environment. This setup should use TLS like everything else. Since we also develop different applications it is a challenge to use wildcard certificates.

Basically we have hundreds of permutations like this:

ahost.computer001.swe.test
bhost.computer001.swe.test
chost.computer002.swe.test
ahost.computer003.swe.test
bhost.computer003.swe.test
...

My goal is to deploy Caddy to each development machine and use it to issue certificates on demand signed by our own Intermediate CA that is trusted in the entire organisation.

I was unable to make it work with ahost.*.swe.test site blocks. Using a single :443 block and utilizing the host matcher does work.

But I’m unsure if this is a legitimate approach or if I’m getting myself into trouble further down the line?

2. Error messages and/or full log output:

Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4149845,"logger":"events","msg":"event","name":"tls_get_certificate","id":"11803697-45c7-40d3-a70e-32e680702ecd","origin":"tls","data":{"client_hello":{"CipherSuites":[6682,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"ahost.lse015.swe.test","SupportedCurves":[23130,4588,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[39578,772,771],"RemoteAddr":{"IP":"172.28.96.1","Port":63433,"Zone":""},"LocalAddr":{"IP":"172.28.97.131","Port":443,"Zone":""}}}}
Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4150639,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"ahost.lse015.swe.test"}
Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4150696,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.lse015.swe.test"}
Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4150717,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.swe.test"}
Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4150765,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*.test"}
Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4150784,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*.*"}
Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4150884,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"172.28.96.1","remote_port":"63433","server_name":"ahost.lse015.swe.test","remote":"172.28.96.1:63433","identifier":"ahost.lse015.swe.test","cipher_suites":[6682,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
Apr 29 14:11:55 LSE015 caddy[3754]: {"level":"debug","ts":1777464715.4151363,"logger":"http.stdlib","msg":"http: TLS handshake error from 172.28.96.1:63433: no certificate available for 'ahost.lse015.swe.test'"}

3. Caddy version:

caddy --version
v2.11.2 h1:iOlpsSiSKqEW+SIXrcZsZ/NO74SzB/ycqqvAIEfIm64=

4. How I installed and ran Caddy:

a. System environment:

Debian 13 inside WSL running on a Windows 11 Host.

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

sudo systemctl cat caddy
# /usr/lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete BROKEN Caddy config:

{
        debug
        pki {
                ca lan_ca {
                        name lan_ca
                        root {
                               format pem_file
                               cert /etc/caddy/ca.crt
                        }
                        intermediate {
                               format pem_file
                               cert /etc/caddy/swe-test-intermediate.crt
                               key /etc/caddy/swe-test-intermediate.key
                        }
                }
        }
        on_demand_tls {
                ask http://localhost:2019/ask
        }
        cert_issuer internal {
                ca lan_ca
        }
}

:2019 {
    handle_path /ask {
        @allowed expression {query.domain}.endsWith(".swe.test")
        respond @allowed 200
        respond 403
    }
}

ahost.*.swe.test {
        tls {
                on_demand
        }
        respond "https on ahost"
}

bhost.*.swe.test {
        tls {
                on_demand
        }
        respond "https on bhost"
}

e. My complete WORKING Caddy config:

{
        debug
        pki {
                ca lan_ca {
                        name lan_ca
                        root {
                               format pem_file
                               cert /etc/caddy/ca.crt
                        }
                        intermediate {
                               format pem_file
                               cert /etc/caddy/swe-test-intermediate.crt
                               key /etc/caddy/swe-test-intermediate.key
                        }
                }
        }
        on_demand_tls {
                ask http://localhost:2019/ask
        }
        cert_issuer internal {
                ca lan_ca
        }
}

:2019 {
    handle_path /ask {
        @allowed expression {query.domain}.endsWith(".swe.test")
        respond @allowed 200
        respond 403
    }
}

:443 {
        tls {
                on_demand
        }

        @ahost host ahost.*.swe.test
        @bhost host bhost.*.swe.test

        handle @ahost {
                respond "https on ahost"
        }

        handle @bhost {
                respond "https on bhost"
        }
}

5. Links to relevant resources: