I updated Caddy to 0.9 and now I can’t start it anymore.
OS is Arch Linux, systemd is version 231.
Here is the log output:
elo 06 17:07:36 hostname systemd[1]: Started Caddy HTTP/2 web server.
elo 06 17:07:36 hostname systemd[1]: Starting Wait for Network to be Configured…
elo 06 17:07:36 hostname systemd-networkd-wait-online[23553]: ignoring: lo
elo 06 17:07:36 hostname sudo[23549]: pam_unix(sudo:session): session closed for user root
elo 06 17:07:36 hostname systemd[23552]: caddy.service: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory
elo 06 17:07:36 hostname systemd[1]: caddy.service: Main process exited, code=exited, status=226/NAMESPACE
elo 06 17:07:36 hostname systemd[1]: caddy.service: Unit entered failed state.
elo 06 17:07:36 hostname systemd[1]: caddy.service: Failed with result ‘exit-code’.
elo 06 17:07:36 hostname systemd[1]: caddy.service: Service hold-off time over, scheduling restart.
I also tried the old systemd service version (from Github history), but it gave the error “listen tcp :80: bind: permission denied” and same for port 443.
The caddy-full-bin AUR package uses the upstream systemd service, which might causes the problem. I’m not sure if it helps, but did you read the migration instructions for the new package?
Thanks for the swift response, Andreas.
Here is the output for all the commands: http://pastebin.ca/3680455
I do not have the user www-data and instead I use “http” like I did with Caddy 0.8 already, adjusting the old systemd service then as well. Thanks for the migration guide, I had not seen it before. Maybe it should be mentioned in the AUR somewhere.
The migration guide seems to be mostly about the user. Is it really necessary for me to switch to www-data? I’d just like to know before I recursively chown my web files to this different user.
Now I did sudo mkdir -p /etc/ssl/caddy && sudo chown -R http:http /etc/ssl/caddy and it seems the namespace failure was related to it, because now I get this failure:
caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument
Sorry for the late reply, I did not receive a notification on your comment.
You can override the run user and group with systemctl like this:
$ systemctl edit caddy
# opens your $EDITOR, probably vim
[Service]
; User and group the process will run as.
User=http
Group=http
# save and exit
$ systemctl daemon-reload && systemctl restart caddy
Caddy should now run as http user.
Nonetheless, you have to make sure that the http user (or under whatever user caddy runs) has permission to access /etc/ssl/caddy and your website root.
UPDATE
I’ve overlooked that you already changed the user and group of you caddy service.
It looks like you have directly edited /usr/lib/systemd/system/caddy.service which I don’t recommend because you will have to make those changes on every package update.
Please reinstall caddy-full-bin pacaur/yaourt -S caddy-full-bin and use systemctl edit to make those changes.
You can try to comment out ProtectHome=true and ProtectSystem=full in /usr/lib/systemd/system/caddy.service and don’t forget to systemctl daemon-reload thereafter.
I still think that caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument this is some kind of permission problem.
Please let me know if you had luck fixing this issue.
I did this, but it didn’t help. The same capabilities error message keeps appearing.
I then proceeded to comment out, one by one
PrivateTmp=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true
Did not help.
What permission confusion could be at play as there have been no changes since the old Caddy?
I don’t know about init systems in general, but boy does systemd seem opaque in its failures. I wonder how many conferences they still have to have to make it better?
Can you run caddy without the service: TMPDIR=$(mktemp -d) && sudo chown http:http $TMPDIR && cd $TMPDIR && sudo -u http -g http caddy?
This should show an 404 on localhost:2015
cd $TMPDIR failed with permission denied, so I turned to root.
Then sudo -u http -g http caddy gave:
[ERROR] Unable to migrate certificate storage: rename /srv/http/.caddy/letsencrypt /srv/http/.caddy/acme/acme-v01.api.letsencrypt.org: directory not empty
So I did sudo mv /srv/http/.caddy/letsencrypt/ /srv/http/.caddy/letsencrypt.old
Tried running caddy again, got:
open Caddyfile: permission denied
This is because I swapped the cd and chown in my command example, the correct version is: TMPDIR=$(mktemp -d) && cd $TMPDIR && sudo chown http:http $TMPDIR && sudo -u http -g http caddy
You should not get the open Caddyfile: permission denied warning, my terminal output looks like this:
$ TMPDIR=$(mktemp -d) && cd $TMPDIR && sudo chown http:http $TMPDIR && sudo -u http -g http caddy
Activating privacy features... done.
http://:2015
WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192".
Yes, because we didn’t specify any site root, therefore caddy returns a 404.
This error was shown because caddy 0.9+ tries to migrate old certificates on the first run but failed, due to the existing target directory.
I’m running out of ideas, could solve the problem in the meantime?
I had exactly this problem while trying to set up Caddy on Arch.
How long has it been since your last reboot? I was trying to figure out how the AmbientCapabilities directive worked (as I saw that in the systemd service file), and I found out that it was added to systemd in February.
This made me suspect that the Caddy service was relying on features added in a newer version than I was running-- so I just rebooted my server and the service works perfectly now. I had an uptime of 279 days, for reference.
Maybe I should ping klingt as well.