1. The problem I’m having:
I’m trying to use GitHub - caddy-dns/route53: Caddy module: dns.providers.route53 to get the certificate from let’s encrypt for servers that are served in an internal network (not accessible outside).
Unfortunately, it seems to get stuck when trying to solve the DNS challenge
2. Error messages and/or full log output:
2024/10/27 13:48:54.435 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc000259500"}
2024/10/27 13:48:54.435 WARN http.auto_https server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server {"server_name": "srv1", "http_port": 80}
2024/10/27 13:48:54.436 INFO http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2024/10/27 13:48:54.436 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2024/10/27 13:48:54.436 DEBUG http.auto_https adjusted config {"tls": {"automation":{"policies":[{"subjects":["home.intern.agiletomato.com"]},{}]}}, "http": {"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.122.202:8123"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":80"],"routes":[{},{"handle":[{"handler":"vars","root":"/usr/share/caddy"},{"handler":"file_server","hide":["/etc/caddy/Caddyfile"]}]},{}],"automatic_https":{"disable":true}}}}}
2024/10/27 13:48:54.436 INFO tls.cache.maintenance stopped background certificate maintenance {"cache": "0xc000259500"}
2024/10/27 13:48:54.737 INFO admin admin endpoint started {"address": "unix//run/caddy/admin.socket", "enforce_origin": false, "origins": ["", "//127.0.0.1", "//::1"]}
2024/10/27 13:48:54.738 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc0003de480"}
2024/10/27 13:48:54.738 INFO http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2024/10/27 13:48:54.738 INFO http.auto_https enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2024/10/27 13:48:54.738 WARN http.auto_https server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server {"server_name": "srv1", "http_port": 80}
2024/10/27 13:48:54.738 DEBUG http.auto_https adjusted config {"tls": {"automation":{"policies":[{"subjects":["home.intern.agiletomato.com"]},{}]}}, "http": {"servers":{"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"192.168.122.202:8123"}]}]}]}],"terminal":true}],"tls_connection_policies":[{}],"automatic_https":{}},"srv1":{"listen":[":80"],"routes":[{},{"handle":[{"handler":"vars","root":"/usr/share/caddy"},{"handler":"file_server","hide":["/etc/caddy/Caddyfile"]}]},{}],"automatic_https":{"disable":true}}}}}
2024/10/27 13:48:54.739 DEBUG http starting server loop {"address": "[::]:80", "tls": false, "http3": false}
2024/10/27 13:48:54.739 INFO http.log server running {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2024/10/27 13:48:54.739 INFO http enabling HTTP/3 listener {"addr": ":443"}
2024/10/27 13:48:54.739 DEBUG http starting server loop {"address": "[::]:443", "tls": true, "http3": true}
2024/10/27 13:48:54.740 INFO http.log server running {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/10/27 13:48:54.740 INFO http enabling automatic TLS certificate management {"domains": ["home.intern.agiletomato.com"]}
2024/10/27 13:48:54.740 ERROR unable to autosave config {"file": "/etc/caddy/autosave.json", "error": "open /etc/caddy/autosave.json: read-only file system"}
2024/10/27 13:48:54.740 INFO serving initial configuration
2024/10/27 13:48:54.741 INFO tls.obtain acquiring lock {"identifier": "home.intern.agiletomato.com"}
2024/10/27 13:48:54.746 INFO tls storage cleaning happened too recently; skipping for now {"storage": "FileStorage:/var/lib/caddy", "instance": "1b227ed4-1825-4a12-966b-872eaaeedc58", "try_again": "2024/10/28 13:48:54.746", "try_again_in": 86399.999998916}
2024/10/27 13:48:54.746 INFO tls finished cleaning storage units
2024/10/27 13:48:54.748 INFO tls.obtain lock acquired {"identifier": "home.intern.agiletomato.com"}
2024/10/27 13:48:54.748 INFO tls.obtain obtaining certificate {"identifier": "home.intern.agiletomato.com"}
2024/10/27 13:48:54.748 DEBUG events event {"name": "cert_obtaining", "id": "cc3ffaa8-3a5b-4704-abb7-96da081d1d7f", "origin": "tls", "data": {"identifier":"home.intern.agiletomato.com"}}
2024/10/27 13:48:54.749 DEBUG tls.obtain trying issuer 1/1 {"issuer": "acme-v02.api.letsencrypt.org-directory"}
2024/10/27 13:48:54.750 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["home.intern.agiletomato.com"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/10/27 13:48:54.750 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["home.intern.agiletomato.com"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2024/10/27 13:48:54.750 INFO tls.issuance.acme using ACME account {"account_id": "https://acme-v02.api.letsencrypt.org/acme/acct/2023152457", "account_contact": []}
2024/10/27 13:48:55.332 DEBUG tls.issuance.acme.acme_client http request {"method": "GET", "url": "https://acme-v02.api.letsencrypt.org/directory", "headers": {"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["746"],"Content-Type":["application/json"],"Date":["Sun, 27 Oct 2024 13:48:55 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/10/27 13:48:55.333 DEBUG tls.issuance.acme.acme_client creating order {"account": "https://acme-v02.api.letsencrypt.org/acme/acct/2023152457", "identifiers": ["home.intern.agiletomato.com"]}
2024/10/27 13:48:55.521 DEBUG tls.issuance.acme.acme_client http request {"method": "HEAD", "url": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "headers": {"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Cache-Control":["public, max-age=0, no-cache"],"Date":["Sun, 27 Oct 2024 13:48:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["XM4ukI6HU3Ommrn9q0a_yT3U2DiWr5gkBlFj7z-Ih9k1j5EL5lw"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/10/27 13:48:55.829 DEBUG tls.issuance.acme.acme_client http request {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/new-order", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2023152457"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["353"],"Content-Type":["application/json"],"Date":["Sun, 27 Oct 2024 13:48:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/2023152457/317529898907"],"Replay-Nonce":["XM4ukI6H6NtkEyDHWp4Tcq0NnLETWQ0RCv-NuaGgHkovEHdOY6A"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 201}
2024/10/27 13:48:56.025 DEBUG tls.issuance.acme.acme_client http request {"method": "POST", "url": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/421919347487", "headers": {"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]}, "response_headers": {"Boulder-Requester":["2023152457"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["811"],"Content-Type":["application/json"],"Date":["Sun, 27 Oct 2024 13:48:55 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["XM4ukI6HbGnt6hq05f5UmJ4TkNeG3HjrWlZsb8RKL0ZqzZfhUSs"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}, "status_code": 200}
2024/10/27 13:48:56.026 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "home.intern.agiletomato.com", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
3. Caddy version:
I’ve installed caddy with xcaddy --with github.com/caddy-dns/route53
caddy version
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
4. How I installed and ran Caddy:
I installed caddy with pacman -Sy caddy
I then used xcaddy to build a new binary with xcaddy --with github.com/caddy-dns/route53 and replaced /usr/bin/caddy with the newly built binary
a. System environment:
EndeavourOS
b. Command:
sudo systemctl enable caddy
c. Service/unit/compose file:
The unit file used was installed by pacman -Sy caddy
# /usr/lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target
StartLimitIntervalSec=14400
StartLimitBurst=10
[Service]
Type=notify
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/etc
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
ExecStopPost=/usr/bin/rm -f /run/caddy/admin.socket
# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal
# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
# Hardening options
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DevicePolicy=closed
LockPersonality=true
MemoryAccounting=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=strict
RemoveIPC=true
ReadWritePaths=/var/lib/caddy /var/log/caddy /run/caddy
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
First file /etc/caddy/CaddyFile
{
# Restrict the admin interface to a local unix file socket whose directory
# is restricted to caddy:caddy. By default the TCP socket allows arbitrary
# modification for any process and user that has access to the local
# interface. If admin over TCP is turned on one should make sure
# implications are well understood.
admin "unix//run/caddy/admin.socket"
log {
output file /var/log/caddy/caddy.log
format console
level DEBUG
}
}
http:// {
# Set this path to your site's directory.
root * /usr/share/caddy
# Enable the static file server.
file_server
# Another common task is to set up a reverse proxy:
# reverse_proxy localhost:8080
# Or serve a PHP site through php-fpm:
# php_fastcgi localhost:9000
# Refer to the directive documentation for more options.
# https://caddyserver.com/docs/caddyfile/directives
}
# Import additional caddy config files in /etc/caddy/conf.d/
import /etc/caddy/conf.d/*
Second file /etc/caddy/conf.d/home
home.intern.agiletomato.com {
tls {
dns route53 {
max_retries 10 # optional
profile "default"
access_key_id "AKI....." # required if $AWS_ACCESS_KEY
secret_access_key "Zj....." # required if $AWS_SECRET_ACCESS_KEY is not defined
region "us-east-1" # required if $AWS_REGION is not defined
max_wait_dur 60, # propagation wait duration in seconds (optional)
wait_for_propagation false # wait for records to propagate (optional)
}
}
reverse_proxy 192.168.122.202:8123
}
5. Additional attempts
I’ve tried running caddy run --config /etc/caddy/CaddyFile
2024/10/27 15:25:56.510 INFO using config from file {"file": "/etc/caddy/Caddyfile"}
2024/10/27 15:25:56.514 INFO adapted config to JSON {"adapter": "caddyfile"}
2024/10/27 15:25:56.515 INFO redirected default logger {"from": "stderr", "to": "/var/log/caddy/caddy.log"}
2024/10/27 15:25:58.676 INFO route53: unable to load AWS SDK config, failed to get shared config profile, default
