Caddy does not start as a service when using `tls internal`

Help
1

1. Caddy version (caddy version):

v.2.4.6

2. How I run Caddy:

systemd

a. System environment:

Manjaro Gnome.

b. Command:

systemctl start caddy.service

c. Service/unit/compose file:

The one came with the distribution package.

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/etc
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
ExecStopPost=/usr/bin/rm -f /run/caddy/admin.socket

# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal

# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=5s

LimitNOFILE=1048576
LimitNPROC=512

# Hardening options
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DevicePolicy=closed
LockPersonality=true
MemoryAccounting=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=strict
RemoveIPC=true
ReadWritePaths=/var/lib/caddy /var/log/caddy /run/caddy
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# https://caddyserver.com/docs/caddyfile
#
# The configuration below serves a welcome page over HTTP on port 80.
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace the line below with your
# domain name.
#
# https://caddyserver.com/docs/caddyfile/concepts#addresses
{
        # Restrict the admin interface to a local unix file socket whose directory
        # is restricted to caddy:caddy. By default the TCP socket allows arbitrary
        # modification for any process and user that has access to the local
        # interface. If admin over TCP is turned on one should make sure
        # implications are well understood.
        admin "unix//run/caddy/admin.socket"
}

git.nas3.noom.home {
        tls internal
        respond "Hello"
}

3. The problem I’m having:

The service does not start.

4. Error messages and/or full log output:

ม.ค. 29 22:04:15 nas3 systemd[1]: Starting Caddy web server...
ม.ค. 29 22:04:15 nas3 caddy[5146]: {"level":"info","ts":1643468655.6806788,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
ม.ค. 29 22:04:15 nas3 caddy[5146]: {"level":"warn","ts":1643468655.6811702,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":13}
ม.ค. 29 22:04:15 nas3 caddy[5146]: {"level":"info","ts":1643468655.6813505,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003eb570"}
ม.ค. 29 22:04:15 nas3 caddy[5146]: {"level":"info","ts":1643468655.681369,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003eb570"}
ม.ค. 29 22:04:15 nas3 caddy[5146]: validate: loading http app module: provision http: getting tls app: loading tls app module: provision tls: provisioning automation pollicy 0: loading TLS automation management module: position 0: loading module 'internal': provision tls.issuance.internal: loading pki app module: provision pki: provisionioning CA 'local': parsing root certificate PEM: no PEM block found
ม.ค. 29 22:04:15 nas3 systemd[1]: caddy.service: Control process exited, code=exited, status=1/FAILURE
ม.ค. 29 22:04:15 nas3 systemd[1]: caddy.service: Failed with result 'exit-code'.
ม.ค. 29 22:04:15 nas3 systemd[1]: Failed to start Caddy web server.

5. What I already tried:

  • Remove the tls internal and it starts.
  • Also running the command directly caddy start with the same config file will start as well.
  • changing the permission of /root/.local/share/caddy/**/ does not help, same error.
  • Run # XDG_DATA_HOME=/var/lib caddy trust as per Caddy - ArchWiki suggestions.

6. Links to relevant resources:

2

How did you install Caddy, exactly? What steps did you take?

When you run caddy start, you’re running it as your current user. When run as a systemd service, it runs as the caddy user (as you can see in your systemd config).

The caddy user’s HOME should be /var/lib/caddy, but it looks like you’re using a different systemd config than the one we recommend in our docs: Keep Caddy Running — Caddy Documentation. Your systemd config sets Environment=XDG_DATA_HOME=/var/lib.

You can see how Caddy determines its storage location on this page: Conventions — Caddy Documentation.

You can clean out/empty the /var/lib/caddy location to get a fresh start. I’m guessing that some of the files in that location got broken, according to that error message.

3

How did you install Caddy, exactly? What steps did you take?

Caddy is installed using the package manager. I’m using Manjaro here and it’s installed through pamac command. It’s essentially a pacman wrapper (Manjaro is Arch Linux-based).

$ sudo pamac install caddy

Also I should add that I have done the $ XDG_DATA_HOME=/var/lib caddy trust command as per suggested by Arch Wiki.

I’ll look at the document you suggested. Thank you for the helps.

4

Oh, in that case the files may have been written with the wrong user’s permissions. Caddy might not be able to read them if they’re owned by some other user.

You should only run that trust command after starting Caddy with tls internal… for now.

We have fixes coming in the next release which will make overriding XDG_DATA_HOME for the caddy trust command unnecessary, via a rewrite of how the caddy trust command works (by using the admin API to fetch the root cert instead of reading it directly from disk)

1 Like
5

I got it working now. Basically I have to change the owner of all file inside /var/lib/caddy/pki to the user caddy. They were owned by root:root before.

So yeah the problem was wrong user’s permissions as you suggested.

Thank you for your helps again.

2 Likes
6

This topic was automatically closed after 30 days. New replies are no longer allowed.