1. Caddy version (caddy version):
v2.4.3
2. How I run Caddy:
Caddy is installed in a VM on my Proxmox server. I have wanted to transition from Nginx for some time. Nginx works fine, certificates are not a problem, everything works great.
Caddy seems simpler and less effort to maintain and tinker with.
I have my Proxmox server running all my services in VM’s. The Proxmox server sits in it’s own VLAN with pfSense as my edge device. I’ve forwarded ports 80 and 443 to my Caddy server (this hasn’t changed since running Nginx).
Caddy is installed as a systemd service and my Caddyfile is at /etc/caddy/Caddyfile.
I’ve recently switched to Cloudflare for my DNS hosting and I feel that this is the issue. I’ve created A records for my subdomains and pointed them to my public static IP. These are “proxied” (orange). I’m using Full (strict) SSL/TTL with HSTS enabled. Something to note that may also be an issue is that I had enabled this when using Nginx and (stupidly_ revoked all the existing certificates for all subdomains when I switched to Cloudflare.
My Nameservers have been set to Cloudflares in my registrar’s control panel.
a. System environment:
Ubuntu Server 20.04 LTS
b. Command:
sudo systemctl restart caddy
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
My complete Caddyfile
{
acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
email me@example.com
}
vault.domain.com:443 {
tls {
dns cloudflare `redacted`
resolvers 1.1.1.1
}
log {
level INFO
output file /var/log/caddy/caddy.log {
roll_size 10MB
roll_keep 10
}
}
encode gzip
header {
Strict-Transport-Security "max-age=31536000; includesubdomains; preload"
X-XSS-Protection "1; mode=block"
X-Frame-Options "DENY"
X-Robots-Tag "none"
-Server
}
@insecureadmin {
not remote_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
path /admin*
}
redir @insecureadmin /
reverse_proxy /notifications/hub 10.128.0.2:3012
reverse_proxy 10.128.0.2:80 {
header_up X-Real-IP {remote_host}
}
}
3. The problem I’m having:
I feel like i’m missing something simple, but I can’t for the life of me get a certificate for my vault.domain.com subdomain. I know it’s not a pfSense issue as Nginx was working just fine and i’m getting nothing in the pfSense logs to suggest that this is the issue.
4. Error messages and/or full log output:
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.045477,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"warn","ts":1629121005.049966,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0618904,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0626307,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0629268,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0649357,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["vault.domain.com"]}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0665667,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Aug 16 21:36:45 caddy systemd[1]: Started Caddy.
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0710037,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000440540"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0721838,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0723846,"logger":"tls","msg":"finished cleaning storage units"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0728822,"logger":"tls.obtain","msg":"acquiring lock","identifier":"vault.domain.com"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0737667,"msg":"serving initial configuration"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.077342,"logger":"tls.obtain","msg":"lock acquired","identifier":"vault.domain.com"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.0875306,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["vault.domain.com"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"me@example.com"}
Aug 16 21:36:45 caddy caddy[6854]: {"level":"info","ts":1629121005.087666,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["vault.domain.com"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"me@example.com"}
Aug 16 21:36:47 caddy caddy[6854]: {"level":"info","ts":1629121007.3360412,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"vault.domain.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Aug 16 21:38:27 caddy caddy[6854]: {"level":"error","ts":1629121107.3394582,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"vault.domain.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for vault.domain.com (probably OK if presenting failed)"}
Aug 16 21:38:27 caddy caddy[6854]: {"level":"error","ts":1629121107.5826492,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"vault.domain.com","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[vault.domain.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.vault.domain.com\": could not find the start of authority for _acme-challenge.vault.domain.com.: dial tcp 1.1.1.1:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/23362238/311344198) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Aug 16 21:38:27 caddy caddy[6854]: {"level":"info","ts":1629121107.5830271,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["vault.domain.com"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"me@example.com"}
Aug 16 21:38:27 caddy caddy[6854]: {"level":"info","ts":1629121107.5830584,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["vault.domain.com"],"ca":"https://acme-staging-v02.api.letsencrypt.org/directory","account":"me@example.com"}
Aug 16 21:38:29 caddy caddy[6854]: {"level":"info","ts":1629121109.9600258,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"vault.domain.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Aug 16 21:40:09 caddy caddy[6854]: {"level":"error","ts":1629121209.96373,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"vault.domain.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for vault.domain.com (probably OK if presenting failed)"}
Aug 16 21:40:10 caddy caddy[6854]: {"level":"error","ts":1629121210.2070925,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"vault.domain.com","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[vault.domain.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.vault.domain.com\": could not find the start of authority for _acme-challenge.vault.domain.com.: dial tcp 1.1.1.1:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/23362238/311355538) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Aug 16 21:40:10 caddy caddy[6854]: {"level":"error","ts":1629121210.2071354,"logger":"tls.obtain","msg":"will retry","error":"[vault.domain.com] Obtain: [vault.domain.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.vault.domain.com\": could not find the start of authority for _acme-challenge.vault.domain.com.: dial tcp 1.1.1.1:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/23362238/311355538) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":205.129645697,"max_duration":2592000}
Aug 16 21:41:12 caddy caddy[6854]: {"level":"info","ts":1629121272.1823723,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"vault.domain.com","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
5. What I already tried:
I’ve double checked my zone API Token in Cloudflare and I’m quite sure i’ve set this up correctly. It’s been almost two days so I’m positive DNS records have propagated.
My Cloudflare API Token is set up like this:
Zone - Zone - Read
Zone - DNS - Edit
Include - Specific Zone - domain.com
I’m at a loss with this, and I’m certain it’s something stupidly simple!