Cannot start Caddy as a systemd service

[Nota (Nota Xiao)]

Hi everybody,

I am using the caddy.service file from https://github.com/mholt/caddy/tree/master/dist/init/linux-systemd. But I can’t start Caddy.

journalctl -xe returns:

Jan 23 12:45:39 150-95-143-145.conoha.io systemd[1]: Network Service is not active.
Jan 23 12:45:39 150-95-143-145.conoha.io systemd[1]: Dependency failed for Wait for Network to be Configured.
-- Subject: Unit systemd-networkd-wait-online.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit systemd-networkd-wait-online.service has failed.
-- 
-- The result is dependency.
Jan 23 12:45:39 150-95-143-145.conoha.io systemd[1]: systemd-networkd-wait-online.service: Job systemd-networkd-wait-online.service/start failed wit
Jan 23 12:45:39 150-95-143-145.conoha.io systemd[1]: caddy.service: Start request repeated too quickly.
Jan 23 12:45:39 150-95-143-145.conoha.io systemd[1]: Failed to start Caddy HTTP/2 web server.
-- Subject: Unit caddy.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit caddy.service has failed.
-- 
-- The result is failed.
Jan 23 12:45:39 150-95-143-145.conoha.io systemd[1]: caddy.service: Failed with result 'start-limit-hit'.

And my Caddyfile is:

nota.moe,
next.nota.moe {
	root /var/www/wordpress
	fastcgi / 127.0.0.1:9000 php
	gzip
	log /var/www/log/nota.moe.log
	rewrite {
	    if {path} not_match ^\/wp-admin
	    to {path} {path}/ /index.php?_url={uri}
	}
}

www.nota.moe {
	redir https://nota.moe{url}
}

adultsonly.nota.moe {
	root /var/www
	browse
	basicauth / *** ***
	log /var/www/log/adultsonly.nota.moe.log
}

dl.nota.moe {
	root /var/www/dl
	log /var/www/log/dl.nota.moe.log
}

pma.nota.moe {
	root /var/www/pma
	fastcgi / 127.0.0.1:9000 php
	basicauth / *** ***
	log /var/www/log/pma.nota.moe.log
}

The interesting thing is, if I execute /usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp directly, Caddy will run properly without any error.

Could you help me?

Thanks.

[Whitestrake (Matthew Fay)]

Looks like Caddy is hung up waiting for systemd-networkd-wait-online.service, which is a service that waits for all links known to systemd-networkd to be configured and returns its result based on whether any links have successfully connected.

Since you’re running Caddy just fine from shell… You’re not using a different network manager like netctl or NetworkManager by any chance?

[Nota (Nota Xiao)]

I am using Fedora, the default network manager may be NetworkManager.
I execute systemctl enable systemd-networkd a moment ago and now Caddy service runs well.Thank you.

[Whitestrake (Matthew Fay)]

If you’re going to enable systemd-networkd, I would recommend ensuring that NetworkManager is disabled and that you don’t have any other services depending specifically on functionality it provides.

An easier solution might be to instead make use of NetworkManager's equivalent, called networkmanager-wait-online.service. You would only need to edit the Caddy unit definition, rather than diverting your system configuration from the default.

[Nota (Nota Xiao)]

Yes, I did try to disable NetworkManager, but after that I couldn’t connect to my server through SSH. So I enabled it and disabled systemd-networkd, then edited my caddy.service:

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target
Wants=network-online.target networkmanager-wait-online.service

[Service]
Restart=on-failure
StartLimitInterval=86400
StartLimitBurst=5

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy

; Always set "-root" to something safe in case it gets forgotten in the Caddyfile.
ExecStart=/usr/local/bin/caddy -quic -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp
ExecReload=/bin/kill -USR1 $MAINPID

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576
; Unmodified caddy is not expected to use more than that.
LimitNPROC=64

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=true
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy

; The following additional security directives only work with systemd v229 or later.
; They further retrict privileges that can be gained by caddy. Uncomment if you like.
; Note that you may have to add capabilities required by any plugins in use.
;CapabilityBoundingSet=CAP_NET_BIND_SERVICE
;AmbientCapabilities=CAP_NET_BIND_SERVICE
;NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

Now it runs well.

[tjgillies (Tyler Gillies)]

It appears like this won’t work if I’m running NetworkManager. Do I have to give up NetworkManager to get the systemd service to run?

Edit: Nevermind I’m dumb the above line is to use NetworkManager

Wants=network-online.target networkmanager-wait-online.service

[wmark (Mark)]

Somewhat related: Better not use NetworkManager on a server. If your server gets its IP address by DHCP then upon renewal of the lease your connection will be down for up to a few seconds. Tracked this down recently on Fedora 25.

(And set [DHCP] → CriticalConnection=true in /etc/systemd/network/XXX.network for the link. )