Cannot start Caddy as a service with systemd (Ubuntu 20.04) · GitHub

arenaud · April 12, 2021, 1:53pm

As a new user, I’d like to create a new topic but I get the following error message when I submit it:

“Sorry, new users can only put 4 links in a post. Remember to enclose Caddyfiles and logs with backticks (```) on empty lines immediately before and after the text - domains posted this way do not count as links.”

I made sure all logs and commands are enclosed in code blocks, therefore leaving only two actual links, but I keep getting the same message.
The content I wanted to post is here: Cannot start Caddy as a service with systemd (Ubuntu 20.04) · GitHub.

francislavoie · April 12, 2021, 3:33pm

1. Caddy version (`caddy version`):

v2.3.0

2. How I run Caddy:

I followed the official instructions to install Caddy on Ubuntu and set it up as a service under system (Install — Caddy Documentation).

I am able to get Caddy to run by itself but not as a systemd service.

a. System environment:

Ubuntu 20.04

b. Command:

sudo systemctl daemon-reload
sudo systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

arnaudrenaud.com {
        redir https://www.arnaudrenaud.com{uri}
}

www.arnaudrenaud.com {
        root * /home/ubuntu/apps/art-portfolio
        file_server
        encode gzip
        log {
                output file /home/ubuntu/logs/www.log
        }
}

webhooks.arnaudrenaud.com {
        reverse_proxy localhost:9000
        log {
                output file /home/ubuntu/logs/webhooks.log
        }
}

3. The problem I’m having:

sudo systemctl start caddy results in:

Job for caddy.service failed because a timeout was exceeded.
See "systemctl status caddy.service" and "journalctl -xe" for details.

The ExecStart command from caddy.service does start Caddy correctly when run directly in the shell.

4. Error messages and/or full log output:

journalctl -xef -u caddy.service:

Apr 12 13:21:13 vps-52918bf9 systemd[1]: Starting Caddy...
-- Subject: A start job for unit caddy.service has begun execution
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit caddy.service has begun execution.
--
-- The job identifier is 4455.
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: caddy.HomeDir=/var/lib/caddy
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: caddy.Version=v2.3.0
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: runtime.GOOS=linux
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: runtime.GOARCH=amd64
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: runtime.Compiler=gc
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: runtime.NumCPU=1
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: runtime.GOMAXPROCS=1
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: runtime.Version=go1.15.6
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: os.Getwd=/
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: LANG=C.UTF-8
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: NOTIFY_SOCKET=/run/systemd/notify
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: HOME=/var/lib/caddy
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: LOGNAME=caddy
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: USER=caddy
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: INVOCATION_ID=a99c86336d714fc79cf882e027a6a1e3
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: JOURNAL_STREAM=9:224723
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.6763065,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.6789289,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.6793787,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.6795354,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.680931,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["webhooks.arnaudrenaud.com","www.arnaudrenaud.com","arnaudrenaud.com"]}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.6814835,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00022b570"}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.684435,"logger":"tls","msg":"cleaned up storage units"}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.7044687,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Apr 12 13:21:13 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233673.7046242,"msg":"serving initial configuration"}
Apr 12 13:22:43 vps-52918bf9 systemd[1]: caddy.service: start operation timed out. Terminating.
Apr 12 13:22:43 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233763.7282844,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
Apr 12 13:22:44 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233764.7288136,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00022b570"}
Apr 12 13:22:45 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233765.229118,"logger":"admin","msg":"stopped previous server"}
Apr 12 13:22:45 vps-52918bf9 caddy[14888]: {"level":"info","ts":1618233765.2291603,"msg":"shutdown done","signal":"SIGTERM"}
Apr 12 13:22:45 vps-52918bf9 systemd[1]: caddy.service: Failed with result 'timeout'.
-- Subject: Unit failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- The unit caddy.service has entered the 'failed' state with result 'timeout'.
Apr 12 13:22:45 vps-52918bf9 systemd[1]: Failed to start Caddy.
-- Subject: A start job for unit caddy.service has failed
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit caddy.service has finished with a failure.
--
-- The job identifier is 4455 and the job result is failed.

5. What I already tried:

Browsed existing topics on Caddy forum mentioning the same problem, tried to change a few config options in caddy.service.
Also tried commands mentioned in this tutorial: How To Host a Website with Caddy on Ubuntu 18.04 | DigitalOcean.

6. Links to relevant resources:

francislavoie · April 12, 2021, 3:38pm

^ pasted above. I see no formatting issues either, so

As a hunch, try removing Type=notify. I think that for this to work, you need latest build of Caddy on the master branch. I think systemd is waiting for Caddy to report that it’s running, but that change was only added in these commits:

github.com/caddyserver/caddy

notify: Send all sd_notify signals from main caddy process (#4060)

committed 08:01PM - 05 Apr 21 UTC

carlwgeorge

+43 -18

Initial sd_notify support was added in #3963, but that sent signals from both c…mdRun and cmdReload. This approach has two drawbacks: - Reloads initiated via the API do not send signals. - The signals are sent from different processes, which requires the `NotifyAccess=exec` directive in the unit file. This change moves the NotifyReloading and NotifyReadiness invocations to Load, which address both of those drawbacks. It also adds a complimentary NotifyStopping method which is invoked from handleStop. All the notify methods are defined in a notify package to avoid an import loop.

If you installed Caddy with the deb repo, this wouldn’t have happened:

arenaud · April 12, 2021, 4:05pm

Thank you.

It works after removing Type=notify from caddy.service.
I did not have to go to the latest commits. I have installed Caddy using the instructions you just linked to:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo apt-key add -
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee -a /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

system · May 12, 2021, 1:54pm

This topic was automatically closed after 30 days. New replies are no longer allowed.