1. Caddy version (caddy version):
v2.4.6 and v2.4.6
2. How I run Caddy:
eastcapitol.us is an external domain. y8s.casa is purely internal.
Dynamic dns forwards my domain test.eastcapitol.us to my home IP.
OPNsense firewall forwards ports 80 and 443 to caddy.y8s.casa.
caddy.y8s.casa’s caddy file reverse_proxies https://cloud.y8s.casa
cloud.y8s.casa (aka test.eastcapitol.us) is running nextcloud per the install instructions elsewhere on this site.
a. System environment:
frontend is debian 10 buster
backend is debian 11 bullseye
backend nextcloud is 23.0.0.10
php 8.0 running fpm
b. Command:
N/A, automated
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddyfile or JSON config:
Frontend Caddy File:
{
debug
email email@example.com
}
caddy.y8s.casa {
acme_server
tls internal
}
test.eastcapitol.us {
reverse_proxy https://cloud.y8s.casa {
header_up Host {http.reverse_proxy.upstream.hostport}
header_up X-Forwarded-Host {host}
}
}
Backend Caddy File:
{
debug
acme_ca https://caddy.y8s.casa/acme/local/directory
acme_ca_root /etc/ssl/certs/root.crt
}
cloud.y8s.casa {
# tls {
# ca https://caddy.y8s.casa/acme/local/directory
# ca_root /etc/ssl/certs/root.crt
# }
root * /var/www/nextcloud
file_server
log {
output file /var/log/caddy/nextcloud.log
format single_field common_log
}
php_fastcgi 127.0.0.1:9000 {
env PATH /bin
}
header {
# enable HSTS
Strict-Transport-Security max-age=31536000;
# from nextcloud hardening guide
}
redir /.well-known/carddav /remote.php/dav 301
redir /.well-known/caldav /remote.php/dav 301
# .htaccess / data / config / ... shouldn't be accessible from outside
@forbidden {
path /.htaccess
path /data/*
path /config/*
path /db_structure
path /.xml
path /README
path /3rdparty/*
path /lib/*
path /templates/*
path /occ
path /console.php
}
respond @forbidden 404
}
3. The problem I’m having:
My nextcloud app started reporting “error uploading, bad gateway” occasionally. Eventually it became a permanent issue.
At this point I can no longer visit test.eastcapitol.us because it gives me a 502 error from any location (home or away or via mobile network).
curl -v shows the same thing. 502 Bad gateway.
4. Error messages and/or full log output:
here is the output when I curl -v https://test.eastcapitol.us:
journalctl -fu caddy on frontend:
Feb 17 14:37:06 iot caddy[17628]: {"level":"debug","ts":1645126626.2382948,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"cloud.y8s.casa:443","request":{"remote_addr":"173.8.14.69:59945","proto":"HTTP/1.1","method":"GET","host":"cloud.y8s.casa:443","uri":"/","headers":{"X-Forwarded-Host":["test.eastcapitol.us"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["173.8.14.69"],"User-Agent":["curl/7.79.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":771,"cipher_suite":49196,"proto":"http/1.1","proto_mutual":true,"server_name":"test.eastcapitol.us"}},"duration":0.031191595,"error":"x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")"}
Feb 17 14:37:06 iot caddy[17628]: {"level":"error","ts":1645126626.2415526,"logger":"http.log.error","msg":"x509: certificate signed by unknown authority (possibly because of \"x509: ECDSA verification failure\" while trying to verify candidate authority certificate \"Caddy Local Authority - 2022 ECC Root\")","request":{"remote_addr":"173.8.14.69:59945","proto":"HTTP/1.1","method":"GET","host":"test.eastcapitol.us","uri":"/","headers":{"User-Agent":["curl/7.79.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":771,"cipher_suite":49196,"proto":"http/1.1","proto_mutual":true,"server_name":"test.eastcapitol.us"}},"duration":0.034522308,"status":502,"err_id":"kpbi7sdpm","err_trace":"reverseproxy.statusError (reverseproxy.go:861)"}
Feb 17 14:37:06 iot caddy[17628]: {"level":"error","ts":1645126626.2425354,"logger":"http.log.access","msg":"handled request","request":{"remote_addr":"173.8.14.69:59945","proto":"HTTP/1.1","method":"GET","host":"test.eastcapitol.us","uri":"/","headers":{"User-Agent":["curl/7.79.1"],"Accept":["*/*"]},"tls":{"resumed":false,"version":771,"cipher_suite":49196,"proto":"http/1.1","proto_mutual":true,"server_name":"test.eastcapitol.us"}},"common_log":"173.8.14.69 - - [17/Feb/2022:14:37:06 -0500] \"GET / HTTP/1.1\" 502 0","duration":0.034522308,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}
simultaneous journalctl -fu caddy on the backend:
Feb 17 14:37:06 cloud caddy[1629]: {"level":"debug","ts":1645126626.2108588,"logger":"tls.handshake","msg":"choosing certificate","identifier":"cloud.y8s.casa","num_choices":1}
Feb 17 14:37:06 cloud caddy[1629]: {"level":"debug","ts":1645126626.211041,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"cloud.y8s.casa","subjects":["cloud.y8s.casa"],"managed":true,"issuer_key":"caddy.y8s.casa-acme-local-directory","hash":"97cf8fafc44f629267c7c1ae14f7e58bd12a15e56db9a391390d9319e616e88c"}
Feb 17 14:37:06 cloud caddy[1629]: {"level":"debug","ts":1645126626.2110798,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["cloud.y8s.casa"],"managed":true,"expiration":1645150671,"hash":"97cf8fafc44f629267c7c1ae14f7e58bd12a15e56db9a391390d9319e616e88c"}
Feb 17 14:37:06 cloud caddy[1629]: {"level":"debug","ts":1645126626.2340019,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.10.10.40:53414: remote error: tls: bad certificate"}
5. What I already tried:
-
I have cleared out the certs on both machines from
frontend:
/var/lib/caddy/.local/share/caddy/pki/authorities/local
/var/lib/caddy/.local/share/caddy/certificates
backend:
/etc/ssl/certs/root.crt -
reloaded caddy and re-copied
root.crtfrom/var/lib/caddy/.local/share/caddy/pki/authorities/localon the front end to/etc/ssl/certson the backend. -
I have fiddled around with nextcloud’s config.php URLs, trusted proxies, overwrites, etc.
-
I have created temporary names using just the caddy “respond” directive (and got nothing back)
-
I have restarted php8.0-fpm and caddy a zillion times and rebooted both machines
Best I can guess is that there is some issue with the backend machine obtaining a correct cert from the front end but I’m not sure what else to do or if perhaps I did not properly eliminate all the existing certs for caddy to renew them.
6. Links to relevant resources:
Final note
There were other things running on the frontend that were able to obtain certs just fine. For example if I had homeassistant running with the following config, it works just great:
test.eastcapitol.us {
reverse_proxy localhost:8123
}
I can also obtain internal-only domain certs all day long using hetzner dns challenges.
Thanks for your help!