1. Caddy version:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
2. How I installed, and run Caddy:
Package source: caddy-2.6.4-1.el8.src.rpm
Caddy is run with its systemd service
a. System environment:
Oracle Linux Server 8.7
Linux 5.4.17-2136.316.7.el8uek.aarch64
systemd 239 (239-68.0.2.el8_7.2)
b. Command:
systemctl start caddy
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
media.example.com {
file_server {
root /srv/media
}
basicauth /file.mp4 {
user pass
}
}
turn.example.com {
reverse_proxy http://localhost:3478
}
matrix.example.com {
reverse_proxy @allow http://localhost:8008
respond @deny 403 {
close
}
@allow {
path /health
path /_matrix/*
path /_synapse/client/*
}
@deny {
path /_matrix/federation
path /_matrix/federation/*
}
}
element.example.com {
@nocache {
path /srv/element/config.json
path /srv/element/i18n
path /srv/element/home
path /srv/element/sites
path /srv/element/index.html
}
header @nocache Cache-Control no-cache
file_server {
root /srv/element
}
}
riot.example.com {
redir https://element.example.com{uri}
}
3. The problem I’m having:
Caddy wouldn’t start after upgrading from 2.6.2 to 2.6.4. Logs showed that it was unable to access some certificate files. I’ve been able to fix it by overriding Caddy’s systemd service file, but maybe this is a bug in Caddy and I’d like to know if it should get reported as a bug to Caddy’s issue tracker.
4. Error messages and/or full log output:
systemd[1]: Starting Caddy...
caddy[266672]: caddy.HomeDir=/var/lib/caddy
caddy[266672]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
caddy[266672]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
caddy[266672]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
caddy[266672]: caddy.Version=v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
caddy[266672]: runtime.GOOS=linux
caddy[266672]: runtime.GOARCH=arm64
caddy[266672]: runtime.Compiler=gc
caddy[266672]: runtime.NumCPU=2
caddy[266672]: runtime.GOMAXPROCS=2
caddy[266672]: runtime.Version=go1.18.9
caddy[266672]: os.Getwd=/
caddy[266672]: LANG=en_US.UTF-8
caddy[266672]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
caddy[266672]: NOTIFY_SOCKET=/run/systemd/notify
caddy[266672]: HOME=/var/lib/caddy
caddy[266672]: LOGNAME=caddy
caddy[266672]: USER=caddy
caddy[266672]: INVOCATION_ID=29faea1c522a45d689788ca48cad3f1b
caddy[266672]: JOURNAL_STREAM=9:995290
caddy[266672]: {"level":"info","ts":1676679113.2927415,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
caddy[266672]: {"level":"info","ts":1676679113.2959077,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//12
7.0.0.1:2019"]}
caddy[266672]: {"level":"info","ts":1676679113.2964733,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0"
,"https_port":443}
caddy[266672]: {"level":"info","ts":1676679113.296501,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
caddy[266672]: {"level":"info","ts":1676679113.296605,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0x400018aa10"}
caddy[266672]: {"level":"info","ts":1676679113.2970934,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
caddy[266672]: {"level":"info","ts":1676679113.2971008,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
caddy[266672]: {"level":"debug","ts":1676679113.2973158,"logger":"http","msg":"starting server loop","address":"[::]:443","tls":true,"http3":true}
caddy[266672]: {"level":"info","ts":1676679113.297334,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
caddy[266672]: {"level":"debug","ts":1676679113.2973638,"logger":"http","msg":"starting server loop","address":"[::]:80","tls":false,"http3":false}
caddy[266672]: {"level":"info","ts":1676679113.2973688,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
caddy[266672]: {"level":"info","ts":1676679113.2973719,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["element.example.com","matrix.example.com","media.example.com","turn.example.com","riot.example.com"]}
caddy[266672]: {"level":"info","ts":1676679113.2974474,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0x400018aa10"}
caddy[266672]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [element.example.com matrix.example.com media.example.com turn.example.com riot.example.com]: automate: manage [element.example.com matrix.example.com media.example.com turn.example.com riot.example.com]: element.example.com: caching certificate: open /var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/element.example.com/element.example.com.key: permission denied
systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
systemd[1]: caddy.service: Failed with result 'exit-code'.
systemd[1]: Failed to start Caddy.
5. What I already tried:
I added SELinux exceptions (with ausearch, audit2allow and semodule), which helped get rid of a few file read denials.
But a different denial persisted:
type=SELINUX_ERR msg=audit(1676679113.258:36862): op=security_bounded_transition seresult=denied oldcontext=system_u:system_r:init_t:s0 newcontext=system_u:system_r:httpd_t:s0
However, after setting PrivateDevices=no through an override of the caddy.service file, there are no errors anymore and Caddy runs without issue.