Permission denied on config files after using custom caddy build

Help
1

1. The problem I’m having:

After switching my domains from Gandi to Infomaniak I’ve look to use the Infomaniak plugin.
This one is not present in Download Caddy, so I’ve used xcaddy to build a custom caddy binary with infomaniak plugin, like following:

xcaddy build --with github.com/caddy-dns/infomaniak

I used ansible with GitHub - caddy-ansible/caddy-ansible: Ansible role for installing and configuring the Caddy web server to install caddy. When I’ve use Gandi, it was working fine. But since I’ve changed the binary to my custom one, I’ve got permission errors but not sure how to look to understand why.

I did small changes of the ansible role, but didn’t change permissions part Comparing caddy-ansible:master...nicolas-brousse:allow-custom-package-url · caddy-ansible/caddy-ansible · GitHub.

2. Error messages and/or full log output:

Apr 01 00:25:08 s3 systemd[1]: Started Caddy HTTP/2 web server.
Apr 01 00:25:09 s3 caddy[62644]: caddy.HomeDir=/home/caddy
Apr 01 00:25:09 s3 caddy[62644]: caddy.AppDataDir=/home/caddy/.local/share/caddy
Apr 01 00:25:09 s3 caddy[62644]: caddy.AppConfigDir=/home/caddy/.config/caddy
Apr 01 00:25:09 s3 caddy[62644]: caddy.ConfigAutosavePath=/home/caddy/.config/caddy/autosave.json
Apr 01 00:25:09 s3 caddy[62644]: caddy.Version=v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
Apr 01 00:25:09 s3 caddy[62644]: runtime.GOOS=linux
Apr 01 00:25:09 s3 caddy[62644]: runtime.GOARCH=amd64
Apr 01 00:25:09 s3 caddy[62644]: runtime.Compiler=gc
Apr 01 00:25:09 s3 caddy[62644]: runtime.NumCPU=1
Apr 01 00:25:09 s3 caddy[62644]: runtime.GOMAXPROCS=1
Apr 01 00:25:09 s3 caddy[62644]: runtime.Version=go1.22.1
Apr 01 00:25:09 s3 caddy[62644]: os.Getwd=/
Apr 01 00:25:09 s3 caddy[62644]: LANG=C.UTF-8
Apr 01 00:25:09 s3 caddy[62644]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Apr 01 00:25:09 s3 caddy[62644]: HOME=/home/caddy
Apr 01 00:25:09 s3 caddy[62644]: LOGNAME=www-data
Apr 01 00:25:09 s3 caddy[62644]: USER=www-data
Apr 01 00:25:09 s3 caddy[62644]: INVOCATION_ID=5680d83d20cc4d8daf216e6914b9a7bb
Apr 01 00:25:09 s3 caddy[62644]: JOURNAL_STREAM=8:641927
Apr 01 00:25:09 s3 caddy[62644]: SYSTEMD_EXEC_PID=62644
Apr 01 00:25:09 s3 caddy[62644]: CADDYPATH=/etc/ssl/caddy
Apr 01 00:25:09 s3 caddy[62644]: INFOMANIAK_API_TOKEN=******************
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.152443,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"warn","ts":1711923909.168673,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":2}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.1720665,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//[::1]:2019","//127.0.0.1:2019","//localhost:2019"]}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.1758814,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.1824896,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.184018,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.1842887,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.1854262,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.1961365,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00044a400"}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"warn","ts":1711923909.2004333,"logger":"tls","msg":"unable to get instance ID; storage clean stamps will be incomplete","error":"open /home/caddy/.local/share/caddy/instance.uuid: permission denied"}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"error","ts":1711923909.2027133,"logger":"tls","msg":"could not clean default/global storage","error":"unable to acquire storage_clean lock: creating lock file: open /home/caddy/.local/share/caddy/locks/storage_clean.lock: permission denied"}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.2033086,"logger":"tls","msg":"finished cleaning storage units"}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.2344823,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.2349117,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["*.s3.home.lab","console.s3.home.lab","s3.home.lab"]}
Apr 01 00:25:09 s3 caddy[62644]: {"level":"info","ts":1711923909.2387476,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00044a400"}
Apr 01 00:25:09 s3 caddy[62644]: Error: loading initial config: loading new config: http app module: start: finalizing automatic HTTPS: managing certificates for [*.s3.home.lab console.s3.home.lab s3.home.lab]: automate: manage [*.s3.home.lab console.s3.home.lab s3.home.lab]: *.s3.home.lab:  caching certificate: open /home/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/wildcard_.s3.home.lab/wildcard_.s3.home.lab.key: permission denied
Apr 01 00:25:09 s3 systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Apr 01 00:25:09 s3 systemd[1]: caddy.service: Failed with result 'exit-code'.
Apr 01 00:25:09 s3 systemd[1]: caddy.service: Scheduled restart job, restart counter is at 5.
Apr 01 00:25:09 s3 systemd[1]: Stopped Caddy HTTP/2 web server.
Apr 01 00:25:09 s3 systemd[1]: caddy.service: Start request repeated too quickly.
Apr 01 00:25:09 s3 systemd[1]: caddy.service: Failed with result 'exit-code'.
Apr 01 00:25:09 s3 systemd[1]: Failed to start Caddy HTTP/2 web server.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

b. Command:

"/usr/local/bin/caddy" run --environ --config "/etc/caddy/Caddyfile"

c. Service/unit/compose file:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
StartLimitIntervalSec=60
StartLimitBurst=5

[Service]
Restart=on-failure

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy

ExecStart="/usr/local/bin/caddy" run --environ --config "/etc/caddy/Caddyfile" 
ExecReload="/usr/local/bin/caddy" reload --config "/etc/caddy/Caddyfile"

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=false
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; ... except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy /var/log/caddy

; The following additional security directives only work with systemd v229 or later.
; They further retrict privileges that can be gained by caddy.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true


; Additional environment variables:

Environment=INFOMANIAK_API_TOKEN=**************

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
  servers {
    trusted_proxies static 172.16.0.1/16
  }
}

s3.home.lab, *.s3.home.lab {
  encode zstd gzip

  log

  handle_errors {
    respond "{err.status_code} {err.status_text}"
  }

  reverse_proxy localhost:9091 {
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote.host}
    header_up X-Forwarded-Port {http.request.port}
  }

  tls devops@home.lab {
    dns infomaniak {env.INFOMANIAK_API_TOKEN}
    resolvers 1.1.1.1
  }
}

console.s3.home.lab {
  encode zstd gzip

  log

  handle_errors {
    respond "{err.status_code} {err.status_text}"
  }

  reverse_proxy localhost:9092 {
    header_up Host {http.request.host}
    header_up X-Real-IP {http.request.remote.host}
    header_up X-Forwarded-Port {http.request.port}
  }

  tls devops@home.lab {
    dns infomaniak {env.INFOMANIAK_API_TOKEN}
    resolvers 1.1.1.1
  }
}

5. Links to relevant resources:

2

That looks suspicious. It shouldn’t be trying to read/write to /home/caddy. I don’t know how the Ansible setup works, but normally we have the caddy user’s HOME as /var/lib/caddy.

See Keep Caddy Running — Caddy Documentation which explains the typical systemd setup (which is automated when installing the apt repo).

Remove this, it’s redundant. Caddy already passes through Host as-is.

Are you sure you need these? Caddy passes through X-Forwarded-For and X-Forwarded-Proto which already covers that, and those are correctly handled with trusted proxies etc.

Are you sure you need this? You only need it if you actually have some other proxy in front of Caddy. What do you have as a proxy?