Ok, another test: I was able to run Caddy successfully as user http without systemd with this command:
sudo CADDYPATH=/etc/ssl/caddy su -s /bin/sh http -c “/usr/bin/caddy -log stdout -agree=true -conf=/etc/caddy/caddy.conf -root=/var/tmp”
The user http is “nologin”, so I had to do it that way. Evidently it is really running as http because without the CADDYPATH it always tried to look for the certs in /srv/http which is the home for http.
So the problem only occurs when running through systemd! I thought I could use strace after figuring out how to run Caddy directly as http, but apparently I would have to strace through systemd, however the heck that works… sigh.
Edit: apparently I can’t use strace as it will drop capabilities.
I think the thing I have to solve is this old problem of not being able to set the caps in the systemd service: Starting with systemd: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory - #9 by Beluga
If I don’t comment out the caps settings, it still fails like in that post.
Edit2: I verified the problem is due to systemd upgrade by downgrading to 231. Now if I run it like in the past (the caps stuff commented out), it works fine.