Starting with systemd: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory

I updated Caddy to 0.9 and now I can’t start it anymore.
OS is Arch Linux, systemd is version 231.

Here is the log output:

elo 06 17:07:36 hostname systemd[1]: Started Caddy HTTP/2 web server.
elo 06 17:07:36 hostname systemd[1]: Starting Wait for Network to be Configured…
elo 06 17:07:36 hostname systemd-networkd-wait-online[23553]: ignoring: lo
elo 06 17:07:36 hostname sudo[23549]: pam_unix(sudo:session): session closed for user root
elo 06 17:07:36 hostname systemd[23552]: caddy.service: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory
elo 06 17:07:36 hostname systemd[1]: caddy.service: Main process exited, code=exited, status=226/NAMESPACE
elo 06 17:07:36 hostname systemd[1]: caddy.service: Unit entered failed state.
elo 06 17:07:36 hostname systemd[1]: caddy.service: Failed with result ‘exit-code’.
elo 06 17:07:36 hostname systemd[1]: caddy.service: Service hold-off time over, scheduling restart.

I also tried the old systemd service version (from Github history), but it gave the error “listen tcp :80: bind: permission denied” and same for port 443.

What should I do next?

Meanwhile I went back to Caddy 0.8.3 and the old systemd service file and now everything works again.

It looks like the new binary maybe doesn’t have permission to bind to low ports? Did you run setcap on it?

Ok, now I tried again with 0.9 and did sudo setcap cap_net_bind_service=+ep
I still get the Failed namespace spawning.

At this point I don’t know what that means (I’m not a systemd user) - sorry. Maybe somebody else can help you.

old file had caddy at /usr/bin/caddy and new file has caddy at /usr/local/bin/caddy so could be it ?

No, the file exists in the correct location when I install with the new package. Old package: https://aur.archlinux.org/packages/caddy/ New package: https://aur.archlinux.org/packages/caddy-full-bin/
It is quite fast for me to switch the packages, so keep the tips coming :slight_smile: Maybe I should ping klingt as well.

@Beluga Please post the output of the following commands:

  • ls -al $(which caddy)
  • pacman -Ql caddy-full-bin
  • journalctl -xeu caddy --priority 5 --no-pager --no-hostname
  • systemctl cat caddy

The caddy-full-bin AUR package uses the upstream systemd service, which might causes the problem. I’m not sure if it helps, but did you read the migration instructions for the new package?

2 Likes

Thanks for the swift response, Andreas.
Here is the output for all the commands: http://pastebin.ca/3680455

I do not have the user www-data and instead I use “http” like I did with Caddy 0.8 already, adjusting the old systemd service then as well. Thanks for the migration guide, I had not seen it before. Maybe it should be mentioned in the AUR somewhere.
The migration guide seems to be mostly about the user. Is it really necessary for me to switch to www-data? I’d just like to know before I recursively chown my web files to this different user.

Now I did sudo mkdir -p /etc/ssl/caddy && sudo chown -R http:http /etc/ssl/caddy and it seems the namespace failure was related to it, because now I get this failure:

caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument

1 Like

Sorry for the late reply, I did not receive a notification on your comment.
You can override the run user and group with systemctl like this:

$ systemctl edit caddy
# opens your $EDITOR, probably vim
[Service]
; User and group the process will run as.
User=http
Group=http
# save and exit
$ systemctl daemon-reload && systemctl restart caddy

Caddy should now run as http user.
Nonetheless, you have to make sure that the http user (or under whatever user caddy runs) has permission to access /etc/ssl/caddy and your website root.

UPDATE
I’ve overlooked that you already changed the user and group of you caddy service.
It looks like you have directly edited /usr/lib/systemd/system/caddy.service which I don’t recommend because you will have to make those changes on every package update.
Please reinstall caddy-full-bin pacaur/yaourt -S caddy-full-bin and use systemctl edit to make those changes.

You can try to comment out ProtectHome=true and ProtectSystem=full in /usr/lib/systemd/system/caddy.service and don’t forget to systemctl daemon-reload thereafter.
I still think that caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument this is some kind of permission problem.
Please let me know if you had luck fixing this issue.

2 Likes

I am watching this thread now, so if you still have problems running caddy as a different user please let me know.

1 Like

I did this, but it didn’t help. The same capabilities error message keeps appearing.

I then proceeded to comment out, one by one
PrivateTmp=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

Did not help.

What permission confusion could be at play as there have been no changes since the old Caddy?

I don’t know about init systems in general, but boy does systemd seem opaque in its failures. I wonder how many conferences they still have to have to make it better?

Can you run caddy without the service: TMPDIR=$(mktemp -d) && sudo chown http:http $TMPDIR && cd $TMPDIR && sudo -u http -g http caddy?
This should show an 404 on localhost:2015

cd $TMPDIR failed with permission denied, so I turned to root.
Then sudo -u http -g http caddy gave:
[ERROR] Unable to migrate certificate storage: rename /srv/http/.caddy/letsencrypt /srv/http/.caddy/acme/acme-v01.api.letsencrypt.org: directory not empty

So I did sudo mv /srv/http/.caddy/letsencrypt/ /srv/http/.caddy/letsencrypt.old
Tried running caddy again, got:
open Caddyfile: permission denied

Hmm… I did get 404 on my main site after that. I guess it is the desired result?

This is because I swapped the cd and chown in my command example, the correct version is: TMPDIR=$(mktemp -d) && cd $TMPDIR && sudo chown http:http $TMPDIR && sudo -u http -g http caddy

You should not get the open Caddyfile: permission denied warning, my terminal output looks like this:

$ TMPDIR=$(mktemp -d) && cd $TMPDIR && sudo chown http:http $TMPDIR && sudo -u http -g http caddy
Activating privacy features... done.
http://:2015
WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192".

Yes, because we didn’t specify any site root, therefore caddy returns a 404.

This error was shown because caddy 0.9+ tries to migrate old certificates on the first run but failed, due to the existing target directory.

I’m running out of ideas, could solve the problem in the meantime?

I had exactly this problem while trying to set up Caddy on Arch.

How long has it been since your last reboot? I was trying to figure out how the AmbientCapabilities directive worked (as I saw that in the systemd service file), and I found out that it was added to systemd in February.

This made me suspect that the Caddy service was relying on features added in a newer version than I was running-- so I just rebooted my server and the service works perfectly now. I had an uptime of 279 days, for reference.

I was at the LibreOffice conference last week so could not reply. No, I did not solve the problem yet.

Ninji: I rebooted after the last system update. Almost immediately after that, I started trying the 0.9 Caddy update.

I got this error message, and the cause was that there was no /etc/ssl/caddy folder.

I don’t know if it’ll help OP, but for me, the fix was: sudo mkdir /etc/ssl/caddy, and then set its permissions.

2 Likes

No, see Starting with systemd: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory - #9 by Beluga

The next problem was caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument

I still have not had time to test. I noticed there is a new revision of the service file: https://github.com/mholt/caddy/commits/master/dist/init/linux-systemd/caddy.service