Starting with systemd: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory

Ok, now I tried again with 0.9 and did sudo setcap cap_net_bind_service=+ep
I still get the Failed namespace spawning.

At this point I don’t know what that means (I’m not a systemd user) - sorry. Maybe somebody else can help you.

old file had caddy at /usr/bin/caddy and new file has caddy at /usr/local/bin/caddy so could be it ?

No, the file exists in the correct location when I install with the new package. Old package: https://aur.archlinux.org/packages/caddy/ New package: https://aur.archlinux.org/packages/caddy-full-bin/
It is quite fast for me to switch the packages, so keep the tips coming :slight_smile: Maybe I should ping klingt as well.

@Beluga Please post the output of the following commands:

  • ls -al $(which caddy)
  • pacman -Ql caddy-full-bin
  • journalctl -xeu caddy --priority 5 --no-pager --no-hostname
  • systemctl cat caddy

The caddy-full-bin AUR package uses the upstream systemd service, which might causes the problem. I’m not sure if it helps, but did you read the migration instructions for the new package?

2 Likes

Thanks for the swift response, Andreas.
Here is the output for all the commands: http://pastebin.ca/3680455

I do not have the user www-data and instead I use “http” like I did with Caddy 0.8 already, adjusting the old systemd service then as well. Thanks for the migration guide, I had not seen it before. Maybe it should be mentioned in the AUR somewhere.
The migration guide seems to be mostly about the user. Is it really necessary for me to switch to www-data? I’d just like to know before I recursively chown my web files to this different user.

Now I did sudo mkdir -p /etc/ssl/caddy && sudo chown -R http:http /etc/ssl/caddy and it seems the namespace failure was related to it, because now I get this failure:

caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument

1 Like

Sorry for the late reply, I did not receive a notification on your comment.
You can override the run user and group with systemctl like this:

$ systemctl edit caddy
# opens your $EDITOR, probably vim
[Service]
; User and group the process will run as.
User=http
Group=http
# save and exit
$ systemctl daemon-reload && systemctl restart caddy

Caddy should now run as http user.
Nonetheless, you have to make sure that the http user (or under whatever user caddy runs) has permission to access /etc/ssl/caddy and your website root.

UPDATE
I’ve overlooked that you already changed the user and group of you caddy service.
It looks like you have directly edited /usr/lib/systemd/system/caddy.service which I don’t recommend because you will have to make those changes on every package update.
Please reinstall caddy-full-bin pacaur/yaourt -S caddy-full-bin and use systemctl edit to make those changes.

You can try to comment out ProtectHome=true and ProtectSystem=full in /usr/lib/systemd/system/caddy.service and don’t forget to systemctl daemon-reload thereafter.
I still think that caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument this is some kind of permission problem.
Please let me know if you had luck fixing this issue.

2 Likes

I am watching this thread now, so if you still have problems running caddy as a different user please let me know.

1 Like

I did this, but it didn’t help. The same capabilities error message keeps appearing.

I then proceeded to comment out, one by one
PrivateTmp=true
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

Did not help.

What permission confusion could be at play as there have been no changes since the old Caddy?

I don’t know about init systems in general, but boy does systemd seem opaque in its failures. I wonder how many conferences they still have to have to make it better?

Can you run caddy without the service: TMPDIR=$(mktemp -d) && sudo chown http:http $TMPDIR && cd $TMPDIR && sudo -u http -g http caddy?
This should show an 404 on localhost:2015

cd $TMPDIR failed with permission denied, so I turned to root.
Then sudo -u http -g http caddy gave:
[ERROR] Unable to migrate certificate storage: rename /srv/http/.caddy/letsencrypt /srv/http/.caddy/acme/acme-v01.api.letsencrypt.org: directory not empty

So I did sudo mv /srv/http/.caddy/letsencrypt/ /srv/http/.caddy/letsencrypt.old
Tried running caddy again, got:
open Caddyfile: permission denied

Hmm… I did get 404 on my main site after that. I guess it is the desired result?

This is because I swapped the cd and chown in my command example, the correct version is: TMPDIR=$(mktemp -d) && cd $TMPDIR && sudo chown http:http $TMPDIR && sudo -u http -g http caddy

You should not get the open Caddyfile: permission denied warning, my terminal output looks like this:

$ TMPDIR=$(mktemp -d) && cd $TMPDIR && sudo chown http:http $TMPDIR && sudo -u http -g http caddy
Activating privacy features... done.
http://:2015
WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with "ulimit -n 8192".

Yes, because we didn’t specify any site root, therefore caddy returns a 404.

This error was shown because caddy 0.9+ tries to migrate old certificates on the first run but failed, due to the existing target directory.

I’m running out of ideas, could solve the problem in the meantime?

I had exactly this problem while trying to set up Caddy on Arch.

How long has it been since your last reboot? I was trying to figure out how the AmbientCapabilities directive worked (as I saw that in the systemd service file), and I found out that it was added to systemd in February.

This made me suspect that the Caddy service was relying on features added in a newer version than I was running-- so I just rebooted my server and the service works perfectly now. I had an uptime of 279 days, for reference.

I was at the LibreOffice conference last week so could not reply. No, I did not solve the problem yet.

Ninji: I rebooted after the last system update. Almost immediately after that, I started trying the 0.9 Caddy update.

I got this error message, and the cause was that there was no /etc/ssl/caddy folder.

I don’t know if it’ll help OP, but for me, the fix was: sudo mkdir /etc/ssl/caddy, and then set its permissions.

2 Likes

No, see Starting with systemd: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory - #9 by Beluga

The next problem was caddy.service: Failed at step CAPABILITIES spawning /usr/local/bin/caddy: Invalid argument

I still have not had time to test. I noticed there is a new revision of the service file: https://github.com/mholt/caddy/commits/master/dist/init/linux-systemd/caddy.service

Now I was forced to try upgrading to Caddy 0.9.3 as my sites stopped working for some reason.

I still got the capabilities error, but it was solved with this: Caddy won't start, strace shows it's waiting for a mutex of some sort - #11 by coderobe
“Commenting out CapabilityBoundingSet AmbientCapabilitiesandNoNewPrivileges in the systemd unit file”

I will open a new topic for inability to renew certs.

No need to open a new topic, I just had to learn how to read journalctl log again. Solved with this: Caddy "listen tcp :443: bind: permission denied" - Server Fault

sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/bin/caddy

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.