1. Caddy version (caddy version):
v2.4.6
2. How I run Caddy:
a. System environment:
I’m running Caddy on Arch Linux inside of a Linode virtual server.
> systemctl --version
systemd 250 (250.4-2-arch)
+PAM +AUDIT -SELINUX -APPARMOR -IMA +SMACK +SECCOMP +GCRYPT +GNUTLS +OPENSSL +ACL +BLKID +CURL +ELFUTILS +FIDO2 +IDN2 -IDN +IPTC +KMOD +LIBCRYPTSETUP +LIBFDISK +PCRE2 -PWQUALITY +P11KIT -QRENCODE +BZIP2 +LZ4 +XZ +ZLIB +ZSTD -BPF_FRAMEWORK +XKBCOMMON +UTMP -SYSVINIT default-hierarchy=unified
[11:44:30]
> uname -a
Linux america 5.16.13-x86_64-linode153 #1 SMP PREEMPT Tue Mar 8 14:16:05 EST 2022 x86_64 GNU/Linux
> neofetch
-` nelson@america
.o+` --------------
`ooo/ OS: Arch Linux x86_64
`+oooo: Host: KVM/QEMU (Standard PC (Q35 + ICH9, 2009) pc-q35-4.1)
`+oooooo: Kernel: 5.16.13-x86_64-linode153
-+oooooo+: Uptime: 14 mins
`/:-:++oooo+: Packages: 201 (pacman)
`/++++/+++++++: Shell: fish 3.4.1
`/++++++++++++++: Resolution: 1024x768
`/+++ooooooooooooo/` Terminal: /dev/pts/0
./ooosssso++osssssso+` CPU: AMD EPYC 7642 (1) @ 2.300GHz
.oossssso-````/ossssss+` GPU: 00:01.0 Vendor 1234 Device 1111
-osssssso. :ssssssso. Memory: 152MiB / 954MiB
:osssssss/ osssso+++.
/ossssssss/ +ssssooo/-
`/ossssso+/:- -:/+osssso+-
`+sso+:-` `.-/+oso:
`++:. `-/+/
.` `/
b. Command:
/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
c. Service/unit/compose file:
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
[11:50:50]
d. My complete Caddyfile or JSON config:
# global options block: https://caddyserver.com/docs/caddyfile/options
{
servers {
protocol {
experimental_http3
}
}
email skyfaller@gmail.com
}
# reusable snippets: https://caddyserver.com/docs/caddyfile/concepts#snippets
(cache-plz) {
@cachedFiles {
path *.jpg *.jpeg *.webp
}
@checkCache {
path *.css
}
header @cachedFiles Cache-Control "public, max-age=31536000, must-revalidate"
header @checkCache Cache-Control no-cache
}
(error-response) {
handle_errors {
respond "{http.error.status_code} {http.error.status_text}"
}
}
(boilerplate) {
encode gzip zstd
file_server
import cache-plz
import error-response
}
# redirect no-www to www
(redir-to-www) {
{args.0} {
redir https://www.{args.0}{uri} permanent
}
}
# start site blocks
# local test page, using local certs
america.maximumethics.dev {
root * /srv/maximumethics.dev/test/public/
import boilerplate
header {
Content-Security-Policy: "default-src 'none'; connect-src 'self'; img-src data:; style-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
X-Content-Type-Options: "nosniff"
}
# https://caddyserver.com/docs/caddyfile/directives/templates
# can't use default double curly braces b/c Ansible uses them
templates {
between (( ))
}
tls internal
}
# test page
test.maximumethics.dev {
root * /srv/maximumethics.dev/test/public/
import boilerplate
header {
Content-Security-Policy: "default-src 'none'; connect-src 'self'; img-src data:; style-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
X-Content-Type-Options: "nosniff"
}
# https://caddyserver.com/docs/caddyfile/directives/templates
# can't use default double curly braces b/c Ansible uses them
templates {
between (( ))
}
}
# handcoded Rhode Island site
www.sunrisepvd.com {
root * /srv/sunrisepvd.com/www/public
import boilerplate
php_fastcgi unix//run/php-fpm/php-fpm.sock
header {
Content-Security-Policy: " default-src 'none'; font-src https://fonts.gstatic.com; frame-src https://www.youtube.com; img-src 'self' https://actionnetwork.org; object-src https://actionnetwork.org/includes/js/; script-src 'self' 'unsafe-inline' https://actionnetwork.org/; style-src 'self' 'unsafe-inline' https://actionnetwork.org/css/ https://fonts.googleapis.com/; frame-ancestors 'none'; base-uri 'none'"
X-Content-Type-Options: "nosniff"
}
}
import redir-to-www sunrisepvd.com
# Hugo sites
www.maximumethics.dev {
root * /srv/maximumethics.dev/www/public
encode gzip zstd
file_server browse
import cache-plz
import error-response
header {
Content-Security-Policy: "default-src 'none'; img-src 'self' https://i.ytimg.com https://skyfaller.goatcounter.com; script-src https://gc.zgo.at/count.v2.js 'unsafe-inline'; style-src 'self' 'unsafe-inline'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
X-Content-Type-Options: "nosniff"
}
}
import redir-to-www maximumethics.dev
# temporarily redirect until we have official content
maximumethics.com,
www.maximumethics.com,
maximumethics.org,
www.maximumethics.org {
redir https://www.maximumethics.dev
}
nelson.pavlosky.net {
root * /srv/pavlosky.net/nelson/public
import boilerplate
header {
Content-Security-Policy: "default-src 'none'; frame-src https://jawns.club; img-src 'self' https://i.ytimg.com; script-src https://jawns.club/embed.js; style-src 'self' 'unsafe-inline'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
X-Content-Type-Options: "nosniff"
}
}
www.oahspesales.com {
root * /srv/oahspesales.com/www/public
import boilerplate
header {
Content-Security-Policy: "default-src 'none'; connect-src 'self'; img-src 'self'; style-src 'self'; frame-ancestors 'none'; base-uri 'none'; form-action 'none'"
X-Content-Type-Options: "nosniff"
}
}
import redir-to-www oahspesales.com
www.robertmcfaddenmd.com {
root * /srv/robertmcfaddenmd.com/www/public
import boilerplate
header {
Content-Security-Policy: "default-src 'none'; font-src https://fonts.gstatic.com https://netdna.bootstrapcdn.com; img-src 'self'; script-src 'self' 'unsafe-inline' https://code.jquery.com/jquery-1.12.4.min.js https://netdna.bootstrapcdn.com/bootstrap/3.0.0/js/bootstrap.min.js https://use.fontawesome.com/releases/v5.3.1/js/all.js; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com/ https://netdna.bootstrapcdn.com/bootstrap/3.0.0/css/; frame-ancestors 'none'; base-uri 'none'; form-action https://formspree.io/mcfaddro@yahoo.com"
X-Content-Type-Options: "nosniff"
}
}
import redir-to-www robertmcfaddenmd.com
# Wordpress sites (personal)
stairball.org,
www.stairball.org {
redir https://www.stairball.club{uri} permanent
}
www.skyfaller.space {
root * /srv/skyfaller.space/static/
import boilerplate
header {
Content-Security-Policy: "default-src 'none'; connect-src 'self' https://embedr.flickr.com/photosets/72157594438128195; font-src 'self'; form-action 'none'; frame-src https://embed.spotify.com https://embed.tumblr.com https://player.vimeo.com https://w.soundcloud.com https://www.youtube.com; img-src 'self' *.flickr.com https://live.staticflickr.com https://secure.gravatar.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' https://assets.tumblr.com/post.js https://embedr.flickr.com/assets/* https://widgets.flickr.com/embedr/embedr-47ad26da5deade67d472950b12c94b6c.js; style-src 'self' 'unsafe-inline'; upgrade-insecure-requests"
X-Content-Type-Options: "nosniff"
}
tls internal
}
import redir-to-www skyfaller.space
static.skyfaller.space {
root * /srv/skyfaller.space/static/
import boilerplate
header {
Content-Security-Policy: "default-src 'none'; connect-src 'self' https://embedr.flickr.com/photosets/72157594438128195; font-src 'self'; form-action 'none'; frame-src https://embed.spotify.com https://embed.tumblr.com https://player.vimeo.com https://w.soundcloud.com https://www.youtube.com; img-src 'self' *.flickr.com https://live.staticflickr.com https://secure.gravatar.com; script-src 'self' 'unsafe-eval' 'unsafe-inline' https://assets.tumblr.com/post.js https://embedr.flickr.com/assets/* https://widgets.flickr.com/embedr/embedr-47ad26da5deade67d472950b12c94b6c.js; style-src 'self' 'unsafe-inline'; upgrade-insecure-requests"
X-Content-Type-Options: "nosniff"
}
3. The problem I’m having:
Caddy is not respecting my tls internal setting for www.skyfaller.space (the live site is served by a different server, I’m trying to do testing). It appears to be trying to ask for certs from Let’s Encrypt etc. This might be related to a problem I was having before: Caddyfile: having individual sites differ from global options
4. Error messages and/or full log output:
Here’s some output from sudo journalctl -u caddy.
Clearly, Caddy is not using internal tls for www.skyfaller.space.
Apr 06 12:04:11 america systemd[1]: Started Caddy.
Apr 06 12:04:11 america caddy[2317]: caddy.HomeDir=/var/lib/caddy
Apr 06 12:04:11 america caddy[2317]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Apr 06 12:04:11 america caddy[2317]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Apr 06 12:04:11 america caddy[2317]: caddy.ConfigAutosavePath=/var/lib/caddy/.local/share/caddy/autosave.json
Apr 06 12:04:11 america caddy[2317]: caddy.Version=v2.4.6
Apr 06 12:04:11 america caddy[2317]: runtime.GOOS=linux
Apr 06 12:04:11 america caddy[2317]: runtime.GOARCH=amd64
Apr 06 12:04:11 america caddy[2317]: runtime.Compiler=gc
Apr 06 12:04:11 america caddy[2317]: runtime.NumCPU=1
Apr 06 12:04:11 america caddy[2317]: runtime.GOMAXPROCS=1
Apr 06 12:04:11 america caddy[2317]: runtime.Version=go1.17.3
Apr 06 12:04:11 america caddy[2317]: os.Getwd=/
Apr 06 12:04:11 america caddy[2317]: LANG=en_US.UTF-8
Apr 06 12:04:11 america caddy[2317]: PATH=/usr/local/sbin:/usr/local/bin:/usr/bin
Apr 06 12:04:11 america caddy[2317]: HOME=/var/lib/caddy
Apr 06 12:04:11 america caddy[2317]: LOGNAME=caddy
Apr 06 12:04:11 america caddy[2317]: USER=caddy
Apr 06 12:04:11 america caddy[2317]: INVOCATION_ID=9528d808c19f454f95cf7beac0015815
Apr 06 12:04:11 america caddy[2317]: JOURNAL_STREAM=8:40289
Apr 06 12:04:11 america caddy[2317]: SYSTEMD_EXEC_PID=2317
Apr 06 12:04:11 america caddy[2317]: {"level":"info","ts":1649261051.1938782,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Apr 06 12:04:11 america caddy[2317]: {"level":"info","ts":1649261051.2172556,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Apr 06 12:04:11 america caddy[2317]: {"level":"info","ts":1649261051.2182891,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Apr 06 12:04:11 america caddy[2317]: {"level":"info","ts":1649261051.218443,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 06 12:04:11 america caddy[2317]: {"level":"info","ts":1649261051.2350612,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00054a1c0"}
Apr 06 12:04:11 america caddy[2317]: {"level":"warn","ts":1649261051.2631552,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Apr 06 12:04:11 america caddy[2317]: 2022/04/06 12:04:11 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
Apr 06 12:04:11 america caddy[2317]: 2022/04/06 12:04:11 define JAVA_HOME environment variable to use the Java trust
Apr 06 12:04:11 america sudo[2327]: pam_systemd_home(sudo:auth): systemd-homed is not available: Unit dbus-org.freedesktop.home1.service not found.
Apr 06 12:04:11 america sudo[2327]: pam_unix(sudo:auth): conversation failed
Apr 06 12:04:11 america sudo[2327]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Apr 06 12:04:12 america sudo[2327]: caddy : user NOT in sudoers ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /etc/ca-certificates/trust-source/anchors/Caddy_Local_Authority_-_2021_ECC_Root_112864344905132184111759759757301348053.crt
Apr 06 12:04:12 america caddy[2317]: {"level":"error","ts":1649261052.9375618,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9380097,"logger":"http","msg":"enabling experimental HTTP/3 listener","addr":":443"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9381537,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.skyfaller.space","america.maximumethics.dev","www.maximumethics.org","stairball.org","www.robertmcfaddenmd.com","www.stairball.org","maximumethics.dev","skyfaller.space","sunrisepvd.com","oahspesales.com","nelson.pavlosky.net","www.maximumethics.dev","www.sunrisepvd.com","static.skyfaller.space","maximumethics.org","robertmcfaddenmd.com","www.oahspesales.com","maximumethics.com","www.maximumethics.com","test.maximumethics.dev"]}
Apr 06 12:04:12 america caddy[2317]: {"level":"warn","ts":1649261052.9389827,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [www.skyfaller.space]: no OCSP server specified in certificate"}
Apr 06 12:04:12 america caddy[2317]: {"level":"warn","ts":1649261052.9393365,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [america.maximumethics.dev]: no OCSP server specified in certificate"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9480083,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.local/share/caddy/autosave.json"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.948181,"msg":"serving initial configuration"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.949719,"logger":"tls.obtain","msg":"acquiring lock","identifier":"skyfaller.space"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9503922,"logger":"tls.obtain","msg":"lock acquired","identifier":"skyfaller.space"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9517772,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["skyfaller.space"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"skyfaller@gmail.com"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9518516,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["skyfaller.space"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"skyfaller@gmail.com"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9542181,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Apr 06 12:04:12 america caddy[2317]: {"level":"info","ts":1649261052.9583554,"logger":"tls","msg":"finished cleaning storage units"}
Apr 06 12:04:13 america caddy[2317]: {"level":"info","ts":1649261053.330728,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"skyfaller.space","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Apr 06 12:04:14 america caddy[2317]: {"level":"error","ts":1649261054.018808,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"skyfaller.space","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
Apr 06 12:04:14 america caddy[2317]: {"level":"error","ts":1649261054.0192752,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"skyfaller.space","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/111136433/77868474430","attempt":1,"max_attempts":3}
Apr 06 12:04:15 america caddy[2317]: {"level":"info","ts":1649261055.1603067,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"skyfaller.space","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Apr 06 12:04:18 america caddy[2317]: {"level":"error","ts":1649261058.3103187,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"skyfaller.space","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Invalid response from https://www.skyfaller.space/.well-known/acme-challenge/YTe0y77VZlJojyfbJq4jCutMIdU521dEwHW3iw23Fac [2600:3c03::f03c:91ff:fe55:64ca]: 404","instance":"","subproblems":[]}}
Apr 06 12:04:18 america caddy[2317]: {"level":"error","ts":1649261058.3108,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"skyfaller.space","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Invalid response from https://www.skyfaller.space/.well-known/acme-challenge/YTe0y77VZlJojyfbJq4jCutMIdU521dEwHW3iw23Fac [2600:3c03::f03c:91ff:fe55:64ca]: 404","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/111136433/77868481030","attempt":2,"max_attempts":3}
Apr 06 12:04:19 america caddy[2317]: {"level":"error","ts":1649261059.7241683,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"skyfaller.space","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[skyfaller.space] solving challenges: skyfaller.space: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/111136433/77868496650) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Apr 06 12:04:19 america caddy[2317]: {"level":"info","ts":1649261059.724533,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["skyfaller.space"],"ca":"https://acme.zerossl.com/v2/DV90","account":"skyfaller@gmail.com"}
Apr 06 12:04:19 america caddy[2317]: {"level":"info","ts":1649261059.724544,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["skyfaller.space"],"ca":"https://acme.zerossl.com/v2/DV90","account":"skyfaller@gmail.com"}
Apr 06 12:04:34 america caddy[2317]: {"level":"info","ts":1649261074.4439697,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"skyfaller.space","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Apr 06 12:05:19 america caddy[2317]: {"level":"info","ts":1649261119.3897142,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
5. What I already tried:
Making sure caddy fmt doesn’t complain. Increase maximum UDP Receive Buffer Size.