Systemctl error

Petar_Peric · February 19, 2019, 6:26pm

I’m trying to start caddy with systemctl so that it can run in the background properly.
I’ve tried various things and read many tutorials but nothing seems to work.
I’ve also read this topic: Starting with systemd: Failed at step NAMESPACE spawning /usr/local/bin/caddy: No such file or directory

I’m using ubuntu 16.04 VPS.

This is the error I get:

     ● caddy.service - Caddy HTTP/2 web server
       Loaded: loaded (/etc/systemd/system/caddy.service; enabled; vendor preset: enabled)
       Active: failed (Result: exit-code) since Mon 2019-02-18 23:53:51 CET; 19h ago
         Docs: https://caddyserver.com/docs
     Main PID: 1164 (code=exited, status=226/NAMESPACE)

    Feb 18 23:53:51 ubuntu systemd[1]: Started Caddy HTTP/2 web server.
    Feb 18 23:53:51 ubuntu systemd[1]: caddy.service: Main process exited, code=exited, status=226/NAMESPACE
    Feb 18 23:53:51 ubuntu systemd[1]: caddy.service: Unit entered failed state.
    Feb 18 23:53:51 ubuntu systemd[1]: caddy.service: Failed with result 'exit-code'.

And this is how my caddy.service file looks like:

    [Unit]
    Description=Caddy HTTP/2 web server
    Documentation=https://caddyserver.com/docs
    After=network-online.target
    Wants=network-online.target systemd-networkd-wait-online.service

    [Service]
    Restart=on-abnormal

    ; User and group the process will run as.
    User=www-data
    Group=www-data

    ; Letsencrypt-issued certificates will be written to this directory.
    Environment=CADDYPATH=/etc/ssl/caddy

    ; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
    ExecStart=/usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp
    ExecReload=/bin/kill -USR1 $MAINPID

    ; Use graceful shutdown with a reasonable timeout
    KillMode=mixed
    KillSignal=SIGQUIT
    TimeoutStopSec=5s

    ; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
    LimitNOFILE=1048576
    ; Unmodified caddy is not expected to use more than that.
    LimitNPROC=512

    ; Use private /tmp and /var/tmp, which are discarded after caddy stops.
    PrivateTmp=true
    ; Use a minimal /dev (May bring additional security if switched to 'true', but it may not work on Raspberry Pi's or other devices, so it has been disabled in this dist.)
    PrivateDevices=false
    ; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
    ProtectHome=true
    ; Make /usr, /boot, /etc and possibly some more folders read-only.
    ProtectSystem=full
    ; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
    ;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
    ReadWriteDirectories=/etc/ssl/caddy

    ; The following additional security directives only work with systemd v229 or later.
    ; They further restrict privileges that can be gained by caddy. Uncomment if you like.
    ; Note that you may have to add capabilities required by any plugins in use.
    ;CapabilityBoundingSet=CAP_NET_BIND_SERVICE
    ;AmbientCapabilities=CAP_NET_BIND_SERVICE
    ;NoNewPrivileges=true

    [Install]
    WantedBy=multi-user.target

Any ideas?

Whitestrake · February 19, 2019, 11:10pm

Hi @Petar_Peric,

Throw us the output of which caddy from the shell, and also try it with PrivateTmp=true commented in the unit file.

Petar_Peric · February 19, 2019, 11:32pm

Here’s the output of which caddy:
/usr/local/bin/caddy

Commenting PrivateTmp=true didn’t make a difference.

Whitestrake · February 19, 2019, 11:37pm

OK, hmmmm…

I’m assuming that /etc/ssl/caddy exists and is writable by www-data user or group.

The only other things that occur to me to change are alternately (AND/OR):

Commenting out all of PrivateTmp, PrivateDevices, ProtectHome, and ProtectSystem
Uncommenting CapabilityBoundingSet, AmbientCapabilities, and NoNewPrivileges

Petar_Peric · February 19, 2019, 11:50pm

These are the permissions of caddy folder in /etc/ssl/, do they look ok to you?
drwxrwx--- 2 root www-data 4096 Feb 19 18:47 caddy

I’ve tried changing the lines you suggested and reloading the systemctl deamon but still get the same error

Whitestrake · February 20, 2019, 12:10am

Yeah, looks good (group rwx for www-data).

What distro are you running it on?

Petar_Peric · February 20, 2019, 12:12am

Ubuntu 16.04

Whitestrake · February 20, 2019, 1:08am

Oh, whoops, you mentioned that in the OP, I should have re-read it.

Yeah, I’m stumped. systemd is black magic to me, and searches for 226/NAMESPACE don’t give a great result other than a bunch of threads we’ve had on it in the past.

If I had time I’d spin up a fresh 16.04 VPS somewhere and go through the process of installing Caddy and its unit file from scratch to check whether it’s a problem inherent with the latest Caddy or unit file.

Petar_Peric · February 21, 2019, 1:23pm

I’ve tried it on a fresh VPS and get the same error, don’t know what I’m doing wrong. Any other ideas?

Also, what would be the next best way to run caddy in the background, as an alternative to systemd?

Whitestrake · February 22, 2019, 12:44am

I use Docker for the vast majority of services I run. It brings containers up at boot if they were up at shutdown, pending what you set the container restart policy to. But it’s very much not just a swap-in for an init system; it’s worth treating as a whole other beast if you’re not familiar with containers.

You could also try downloading Caddy with the service plugin and see if the unit file it installs works for you: https://caddyserver.com/docs/hook.service

Petar_Peric · February 22, 2019, 11:41pm

Running with the service plugin worked, thanks!

system · May 23, 2019, 11:41pm

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.