Systemctl failes to start caddy.service due to permission error, but im able to start using caddy start when in etc/caddy

1. The problem I’m having:

systemctl failes to start caddy.service, I’m able to start using systemctl caddy-api.service

I am ONLY able to launch caddy using caddy start when parked in the etc/caddy/ directory
[root@sasuke caddy]# caddy start

[root@sasuke caddy]# pwd
/etc/caddy 

2. Error messages and/or full log output:

[root@sasuke /]# cd etc/caddy
[root@sasuke caddy]# caddy start
2024/03/19 06:27:26.569 INFO    using adjacent Caddyfile
2024/03/19 06:27:26.570 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//[::1]:2019", "//127.0.0.1:2019", "//localhost:2019"]}
2024/03/19 06:27:26.570 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2024/03/19 06:27:26.570 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2024/03/19 06:27:26.570 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0000d9f00"}
2024/03/19 06:27:26.573 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2024/03/19 06:27:26.573 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/03/19 06:27:26.573 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/03/19 06:27:26.573 INFO    http    enabling automatic TLS certificate management   {"domains": ["amp.whitesea.cloud"]}
2024/03/19 06:27:26.575 WARN    tls     storage cleaning happened too recently; skipping for now        {"storage": "FileStorage:/root/.local/share/caddy", "instance": "a78f0b5a-d4ea-43b4-8dfe-0a196c502f9d", "try_again": "2024/03/20 06:27:26.575", "try_again_in": 86399.999999798}
2024/03/19 06:27:26.575 INFO    tls     finished cleaning storage units
2024/03/19 06:27:26.713 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2024/03/19 06:27:26.713 INFO    serving initial configuration
Successfully started Caddy (pid=15382) - Caddy is running in the background
[root@sasuke caddy]# sudo systemctl enable --now caddy.service
Job for caddy.service failed because the control process exited with error code.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.
[root@sasuke caddy]# systemctl status caddy.service
× caddy.service - Caddy
     Loaded: loaded (/etc/systemd/system/caddy.service; enabled; preset: disabled)
     Active: failed (Result: exit-code) since Tue 2024-03-19 02:27:49 EDT; 16s ago
       Docs: https://caddyserver.com/docs/
    Process: 15488 ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile (code=exited, status=1/FAILURE)
   Main PID: 15488 (code=exited, status=1/FAILURE)
        CPU: 31ms

Mar 19 02:27:49 sasuke.whitesea.cloud caddy[15488]: HOME=/var/lib/caddy
Mar 19 02:27:49 sasuke.whitesea.cloud caddy[15488]: LOGNAME=caddy
Mar 19 02:27:49 sasuke.whitesea.cloud caddy[15488]: USER=caddy
Mar 19 02:27:49 sasuke.whitesea.cloud caddy[15488]: INVOCATION_ID=8c1aeedd984f45fca1c314ba15d90da4
Mar 19 02:27:49 sasuke.whitesea.cloud caddy[15488]: JOURNAL_STREAM=8:72793
Mar 19 02:27:49 sasuke.whitesea.cloud caddy[15488]: SYSTEMD_EXEC_PID=15488
Mar 19 02:27:49 sasuke.whitesea.cloud caddy[15488]: Error: reading config file: open /etc/caddy/Caddyfile: permission denied
Mar 19 02:27:49 sasuke.whitesea.cloud systemd[1]: caddy.service: Main process exited, code=exited, status=1/FAILURE
Mar 19 02:27:49 sasuke.whitesea.cloud systemd[1]: caddy.service: Failed with result 'exit-code'.
Mar 19 02:27:49 sasuke.whitesea.cloud systemd[1]: Failed to start Caddy.
[root@sasuke caddy]# getfacl Caddyfile
# file: Caddyfile
# owner: caddy
# group: root
user::rwx
user:caddy:rwx
group::r--
mask::rwx
other::rwx

[root@sasuke caddy]# sudo systemctl daemon-reload
[root@sasuke caddy]# sudo systemctl enable --now caddy
Job for caddy.service failed because the control process exited with error code.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.
[root@sasuke caddy]# sudo useradd --system \
    --gid caddy \
    --create-home \
    --home-dir /var/lib/caddy \
    --shell /usr/sbin/nologin \
    --comment "Caddy web server" \
    caddy
useradd: user 'caddy' already exists
[root@sasuke caddy]# journalctl -u caddy --no-pager | less +G
[root@sasuke caddy]# journalctl -u caddy --no-pager | less +G
[root@sasuke caddy]# caddy version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=
[root@sasuke caddy]#

3. Caddy version: v2.7.6

4. How I installed and ran Caddy:

Installed caddy via the official directions for fedore/cent/rhel.
dnf install caddy

a. System environment:

running centos 9

c. Service/unit/compose file:

[root@sasuke caddy]# cat /etc/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

amp.whitesea.com {
        @proxyamp {
                not path /shared/*
        }
        reverse_proxy @proxyamp localhost:8080

        route /shared/* {
                root * /opt/cubecoders/amp/shared/WebRoot/
                uri strip_prefix /shared
                file_server
        }

        handle_errors {
                @502 {
                        expression {http.error.status_code} == 502
                }
                root * /opt/cubecoders/amp/shared/WebRoot
                rewrite @502 /NotRunning.html
                file_server
        }
}

I have tried changing the owner of the file to caddy as well as using chmod -R 777 for the folder. I’m at a loss. I did try to disable SELinux using setenforce 0 as well, but no dice.

I’d like for caddy to start working at boot, as I have to run caddy start manually every time

You shouldn’t do this, it doesn’t interact with the services at all. This runs Caddy in the background, detached from systemd.

Make sure to fully stop all Caddy instances before trying to start the systemd service. Don’t use caddy start at all after that.

Follow these instructions:

The Caddyfile should be owned by the caddy user. Make sure its ownership is correct. You can fix it with chown caddy:caddy /etc/caddy/Caddyfile. It should not be 777 permissions. You can change it back using chmod 644 /etc/caddy/Caddyfile.

2 Likes

Hey Francis,

Appreciate the reply. I know I did a lot of things that shouldn’t be the norm, but such is the nature of trying to fix your own problems lol

I initially fixed the issue literally minutes before seeing your reply by doing sudo chown -R root:root /etc/caddy/ then doing sudo chmod 755 /etc/caddy from this thread from 2019 Permissions of Caddyfile - #2 by Whitestrake

Note: I beleive my error came from initially only doing chown -R root <filepath> as opposed to root:root. Just linux noob problems

I corrected the ownership to the caddy user according to your instructions and I can confirm it works as it should now.

Thank you for your help!

2 Likes