1. Caddy version (caddy version):
v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=
2. How I run Caddy:
a. System environment:
- Ubuntu 18.04.5
- Go 1.15.8
- xcaddy 0.1.7
b. Command:
./xcaddy build v2.3.0 --with github.com/caddy-dns/cloudflare@latest
sudo mv caddy /usr/bin/
sudo groupadd --system caddy
sudo useradd --system \
--gid caddy \
--create-home \
--home-dir /var/lib/caddy \
--shell /usr/sbin/nologin \
--comment 'Caddy web server' \
caddy
sudo systemctl daemon-reload
sudo systemctl enable caddy
sudo systemctl start caddy
c. Service/unit/compose file:
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile or caddy.json next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/caddy.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddyfile or JSON config:
{
"apps": {
"tls": {
"automation": {
"policies": [{
"subjects": ["*.tryhexadecimal.com"],
"issuers": [{
"module": "acme",
"email": "letsencrypt@tryhexadecimal.com",
"challenges": {
"dns": {
"provider": {
"name": "cloudflare",
"api_token": "API KEY"
}
}
}
}]
},
{
"issuers": [{
"module": "acme",
"email": "letsencrypt@tryhexadecimal.com"
}],
"on_demand": true
}],
"on_demand": {
"ask": "WEBHOOK URL"
}
}
},
"http": {
"servers": {
"srv0": {
"listen": [
":443"
],
"routes": [
{
"match": [
{
"host": [
"*.tryhexadecimal.com",
"*.*.*"
]
}
],
"handle": [
{
"handler": "subroute",
"routes": [
{
"handle": [
{
"encodings": {
"gzip": {}
},
"handler": "encode"
},
{
"handler": "reverse_proxy",
"upstreams": [
{
"dial": "0.0.0.0:3000"
}
]
}
]
}
]
}
],
"terminal": true
}
]
}
}
}
}
}
3. The problem I’m having:
I was trying to upgrade Caddy from 2.0.0.beta15 to 2.3.0 on a production server. Caddy has been running for several months now, and I already had certificates on the disk (both for my domain and third-party domains).
I installed a new version of Go, swapping the old binary with a newer one. I built Caddy using xcaddy with the Cloudflare plugin and swapped Caddy binary as well. The update went smoothly.
Caddy re-issued the wildcard certificate for my domain (which is fine). Caddy has also re-issued certificates for third-party domains, but this time, they were signed by the Caddy Local Authority - ECC Intermediate, not by Let’s Encrypt.
When I tried visiting one of those websites, I got an “untrusted certificate” error. I didn’t understand why all of a sudden, Caddy started issuing certificates using its own CA and not Let’s Encrypt.
4. Error messages and/or full log output:
There weren’t any errors per se. My problem is that Caddy didn’t opt for valid Let’s Encrypt certificates stored in acme-v02.api.letsencrypt.org-directory. Instead, it was serving certificates from the local directory, signed by its own CA (which is untrusted by major browsers).
root@ubuntu $ ls -alF /var/lib/caddy/.local/share/caddy/certificates
drwx------ 4 caddy caddy 4096 Feb 8 09:24 ./
drwx------ 7 caddy caddy 4096 Feb 8 09:30 ../
drwx------ 3 caddy caddy 4096 Feb 8 09:24 acme-v02.api.letsencrypt.org-directory/
drwx------ 13 caddy caddy 4096 Feb 8 08:40 local/
5. What I already tried:
After seeing the “Caddy can’t install a root certificate” in logs, I tried to install Caddy’s root certificate manually. I managed to install a root cert, but since Caddy’s CA isn’t trusted by major browsers, those websites were still inaccessible.
I noticed that Caddy has issued and stored new certificates in a local directory (using its own CA), and was serving certs from there, instead of acme-v02.api.letsencrypt.org-directory. Since I couldn’t find a way of forcing Caddy to serve certificates stored in acme-v02.api.letsencrypt.org-directory, I renamed that directory to local and restarted the server. It worked! But that felt like a hack.
Questions:
- Why Caddy started using its own CA after the upgrade?
- Can I set acme/Let’s Encrypt as a default?
- Is Caddy going to use its own CA when obtaining new certificates again? If that’s the case, then some of my customers’ websites will go down, as no browser trusts Caddy’s CA.
Thanks!
Right now you’re asking Caddy to do something that can’t be done, so Caddy is compensating by signing the certificate itself.