Hi everyone.
I use Caddy both at work and at home as a simple and neat reverse-proxy for all sort of services aimed at the outside world, and it work great! However, I have a problem at home, and I will probably get the same problem at work as we will use a similar setup for internal-only services.
1. The problem I’m having:
I use FreeIPA’s built-in Dogtag instance as my central certificate authority. Since every computer registered with FreeIPA automatically trusts it, it’s an easy way to get a consistent experience without having to deal with self-signed certificates.
For me, the quickest and easiest way to integrate all internal services is to simply grab the certificates from Dogtag through ACME. At work we have some services running like this with Certbot as the ACME client, but I plan to use Caddy as a reverse proxy both at home and at work.
The problem is that it doesn’t seem to work properly.
2. Error messages and/or full log output:
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh systemd[1]: Starting Caddy...
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: caddy.HomeDir=/var/lib/caddy
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: caddy.Version=v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: runtime.GOOS=linux
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: runtime.GOARCH=amd64
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: runtime.Compiler=gc
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: runtime.NumCPU=4
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: runtime.GOMAXPROCS=4
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: runtime.Version=go1.20.10
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: os.Getwd=/
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: LANG=en_US.UTF-8
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: NOTIFY_SOCKET=/run/systemd/notify
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: HOME=/var/lib/caddy
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: LOGNAME=caddy
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: USER=caddy
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: INVOCATION_ID=90d3b0941b264994b4902e1f543b45f4
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: JOURNAL_STREAM=8:36872
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: SYSTEMD_EXEC_PID=3200
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7795296,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"warn","ts":1702415450.77979,"msg":"No files matching import glob pattern","pattern":"Caddyfile.d/*.caddyfile"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"warn","ts":1702415450.7818258,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":6}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7844527,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.784775,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7848241,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7848396,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0001cf680"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415450.7848988,"logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"key_type":"rsa4096"}]}},"http":{"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"/usr/share/caddy"},{"handler":"file_server","hide":["/etc/caddy/Caddyfile"]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}}}}}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7855465,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7858078,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415450.7866764,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7867446,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415450.7869031,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7869391,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.786951,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.internal.thedragonden.ovh"]}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7874804,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7878528,"msg":"serving initial configuration"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh systemd[1]: Started Caddy.
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.7902746,"logger":"tls.obtain","msg":"acquiring lock","identifier":"test.internal.thedragonden.ovh"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"warn","ts":1702415450.8009925,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"e5f08c53-0fd9-408e-b325-874a9d712c67","try_again":1702501850.8009887,"try_again_in":86399.999998817}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.8011577,"logger":"tls","msg":"finished cleaning storage units"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.8020587,"logger":"tls.obtain","msg":"lock acquired","identifier":"test.internal.thedragonden.ovh"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415450.8022566,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"test.internal.thedragonden.ovh"}
Dec 12 22:10:50 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415450.80231,"logger":"events","msg":"event","name":"cert_obtaining","id":"82f88536-10ab-49c2-8276-0f7e5bd6a01f","origin":"tls","data":{"identifier":"test.internal.thedragonden.ovh"}}
Dec 12 22:10:54 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415454.8281102,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"vanadon.internal.thedragonden.ovh-acme-directory"}
Dec 12 22:10:54 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415454.895572,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://vanadon.internal.thedragonden.ovh/acme/directory","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Content-Type":["application/json"],"Date":["Tue, 12 Dec 2023 21:10:54 GMT"],"Server":["Apache/2.4.57 (Rocky Linux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9"],"Vary":["Accept-Encoding"]},"status_code":200}
Dec 12 22:10:54 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415454.9048133,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://vanadon.internal.thedragonden.ovh/acme/new-nonce","headers":{"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["no-transform, no-store"],"Date":["Tue, 12 Dec 2023 21:10:54 GMT"],"Link":["<https://vanadon.internal.thedragonden.ovh/acme/directory>; rel=\"index\""],"Replay-Nonce":["WBaFUbTG3xYZO1sKaplXRA"],"Server":["Apache/2.4.57 (Rocky Linux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9"]},"status_code":200}
Dec 12 22:10:54 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415454.9168513,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://vanadon.internal.thedragonden.ovh/acme/new-account","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.7.6 CertMagic acmez (linux; amd64)"]},"response_headers":{"Content-Length":["131"],"Content-Type":["application/problem+json"],"Date":["Tue, 12 Dec 2023 21:10:54 GMT"],"Server":["Apache/2.4.57 (Rocky Linux) OpenSSL/3.0.7 mod_auth_gssapi/1.6.3 mod_wsgi/4.7.1 Python/3.9"]},"status_code":400}
Dec 12 22:10:54 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"error","ts":1702415454.9171972,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test.internal.thedragonden.ovh","issuer":"vanadon.internal.thedragonden.ovh-acme-directory","error":"HTTP 0 urn:ietf:params:acme:error:badSignatureAlgorithm - Signature of type ES256 not supported\nTry again with RS256."}
Dec 12 22:10:54 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"debug","ts":1702415454.9172792,"logger":"events","msg":"event","name":"cert_failed","id":"197a17a6-2e39-486e-8532-ef86465497fa","origin":"tls","data":{"error":{},"identifier":"test.internal.thedragonden.ovh","issuers":["vanadon.internal.thedragonden.ovh-acme-directory"],"renewal":false}}
Dec 12 22:10:54 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"error","ts":1702415454.917338,"logger":"tls.obtain","msg":"will retry","error":"[test.internal.thedragonden.ovh] Obtain: registering account [] with server: attempt 1: https://vanadon.internal.thedragonden.ovh/acme/new-account: HTTP 0 urn:ietf:params:acme:error:badSignatureAlgorithm - Signature of type ES256 not supported\nTry again with RS256.","attempt":1,"retrying_in":60,"elapsed":4.115234991,"max_duration":2592000}
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415456.6156507,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"warn","ts":1702415456.6159682,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415456.6160536,"logger":"http","msg":"servers shutting down with eternal grace period"}
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh systemd[1]: Stopping Caddy...
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415456.6167154,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh caddy[3200]: {"level":"info","ts":1702415456.616746,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh systemd[1]: caddy.service: Deactivated successfully.
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh systemd[1]: Stopped Caddy.
Dec 12 22:10:56 ouroboros.web-int.celestia.internal.thedragonden.ovh systemd[1]: caddy.service: Consumed 4.235s CPU time.
3. Caddy version:
I tried 2 versions with the same result:
→ Caddy v2.6.4 from EPEL ( caddy-2.6.4-2.el9 | Build Info | koji )
→ Caddy v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A= from Fedora’s COPR ( @caddy/caddy Copr )
4. How I installed and ran Caddy:
a. System environment:
My home domain is thedragonden.ovh.
My FreeIPA server at home is vanadon.internal.thedragonden.ovh. This is not a publicly resolvable domain.
My Caddy server is ouroboros.web-int.celestia.internal.thedragonden.ovh. This is not a publicly resolvable domain.
ouroboros.web-int.celestia.internal.thedragonden.ovh is resolvable by FreeIPA’s internal DNS server (who serve all my home) and I just want to get a certificate from Dogtag for this internal domain as a test.
FreeIPA and Caddy are installed on two separate Rocky Linux 9.2 x64 as RPM packages. No docker, no podman, no intermediary other than my opnSense router. The two servers are on different VLANs but I doubt that is part of the problem. Ports 80 and 443 are open in both directions.
Caddy 2.7.6 was installed with the following commands:
dnf install 'dnf-command(copr)'
dnf copr enable @caddy/caddy
dnf install caddy
b. Command:
The RPM packages come with a systemd service, so I use that to start and stop Caddy:
sudo systemctl start caddy
sudo systemctl stop caddy
c. Service/unit/compose file:
This is the default systemd service ( /usr/lib/systemd/system/caddy.service )
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
It’s as simple as possible:
# The Caddyfile is an easy way to configure your Caddy web server.
#
# https://caddyserver.com/docs/caddyfile
#This this is Caddy's global conf bloc
{
#Turn debug mode ON :
debug
#Tell Caddy to use FreeIPA as an ACME server
acme_ca https://vanadon.internal.thedragonden.ovh/acme/directory
#Try to use an RSA Algorithm as suggested in a 2021 post
# https://caddy.community/t/need-to-understand-what-caddy-tries-to-do-here/12864
key_type rsa4096
}
# The configuration below serves a welcome page over HTTPS on port 443 + 80 as a redirect
https://test.internal.thedragonden.ovh {
# Set this path to your site's directory.
root * /usr/share/caddy
# Enable the static file server.
file_server
}
# As an alternative to editing the above site block, you can add your own site
# block files in the Caddyfile.d directory, and they will be included as long
# as they use the .caddyfile extension.
import Caddyfile.d/*.caddyfile
5. Links to relevant resources:
I guess the relevant error is this:
https://vanadon.internal.thedragonden.ovh/acme/new-account: HTTP 0 urn:ietf:params:acme:error:badSignatureAlgorithm - Signature of type ES256 not supported\nTry again with RS256.
Someone had a similar problem in a similar setup in 2021:
I tried the suggested key_type directive with all possible values, but I get the same results.
FreeIPA, Dogtag and ACME page in the FreeIPA documentation:
https://www.freeipa.org/page/V4/ACME
To enable the FreeIPA ACME service on vanadon.internal.thedragonden.ovh, I used the following command:
ipa-acme-manage enable
I don’t know where the problem lies. I’ve used a similar setup at work with certbot and it works wonderfully, but it seems Caddy and FreeIPA/Dogtag don’t work well together out of the box. Can anyone point me in the right direction?