[Solved] Caddy permission denied when running as systemd after manual installation

[henryfung3a27]

1. The problem I’m having:

I have just manually installed caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=. I picked the caddy_2.8.4_linux_amd64.tar.gz from Github packages and there is simply a caddy executable inside.

I followed the instructions from the manual installation documentation.

I have run

$ sudo setcap cap_net_bind_service=+ep $(which caddy)

In the error messages, there are these specific lines:

Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: caddy.service: Failed to locate executable /usr/bin/caddy: Permission denied
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: caddy.service: Failed at step EXEC spawning /usr/bin/caddy: Permission denied
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Child 3169 belongs to caddy.service.
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Main process exited, code=exited, status=203/EXEC
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Failed with result 'exit-code'.

However, I have eXecutable right for all users.

$ ls -al /usr/bin/caddy
-rwxr-xr-x. 1 root root 40681624 Jun  2 20:10 /usr/bin/caddy

And I can manually run caddy as the caddy user using

$ sudo -u caddy whoami
caddy

$ sudo -u caddy /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
2024/09/25 11:26:22.449	INFO	using config from file	{"file": "/etc/caddy/Caddyfile"}
2024/09/25 11:26:22.449	INFO	adapted config to JSON	{"adapter": "caddyfile"}
2024/09/25 11:26:22.450	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/09/25 11:26:22.451	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc00035fe80"}
2024/09/25 11:26:22.452	WARN	tls	stapling OCSP	{"error": "no OCSP stapling for [hk]: no OCSP server specified in certificate"}
2024/09/25 11:26:22.452	INFO	http.auto_https	automatic HTTP->HTTPS redirects are disabled	{"server_name": "srv0"}
2024/09/25 11:26:22.452	INFO	http	enabling HTTP/3 listener	{"addr": ":8000"}
2024/09/25 11:26:22.452	INFO	failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 7168 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2024/09/25 11:26:22.452	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/09/25 11:26:22.452	INFO	autosaved config (load with --resume flag)	{"file": "/var/lib/caddy/.config/caddy/autosave.json"}
2024/09/25 11:26:22.452	INFO	serving initial configuration
2024/09/25 11:26:22.453	INFO	tls	storage cleaning happened too recently; skipping for now	{"storage": "FileStorage:/var/lib/caddy/.local/share/caddy", "instance": "89d4c866-0f96-46f2-a41f-8eda924f30a4", "try_again": "2024/09/26 11:26:22.453", "try_again_in": 86399.999999803}
2024/09/25 11:26:22.453	INFO	tls	finished cleaning storage units

$ [user@localhost ~]$ curl -k -D- https://localhost:8000
HTTP/2 200 
alt-svc: h3=":8000"; ma=2592000
content-type: text/html; charset=utf-8
date: Wed, 25 Sep 2024 11:28:53 GMT
etag: W/"c9-tkH+OagGKX3eLy85A53ATPJW+TE"
server: Caddy
x-powered-by: Express
content-length: 201

...

2. Error messages and/or full log output:

$ journalctl -u caddy --no-pager | less +G

Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Trying to enqueue job caddy.service/restart/replace
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Installed new job caddy.service/restart as 779
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Enqueued job caddy.service/restart as 779
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Job 779 caddy.service/restart finished, result=done
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Converting job caddy.service/restart -> caddy.service/start
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Passing 0 fds to service
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: About to execute /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Forked /usr/bin/caddy as 3169
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Changed failed -> start
Sep 25 19:17:12 localhost.localdomain systemd[1]: Starting Caddy...
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: User lookup succeeded: uid=982 gid=982
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Bind-mounting / on /run/systemd/unit-root (MS_BIND|MS_REC "")...
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/boot
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/efi
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/etc
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Bind-mounting /run/systemd/unit-root/etc on /run/systemd/unit-root/etc (MS_BIND|MS_REC "")...
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Successfully mounted /run/systemd/unit-root/etc to /run/systemd/unit-root/etc
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/run/credentials
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Bind-mounting /run/systemd/inaccessible/dir on /run/systemd/unit-root/run/credentials (MS_BIND|MS_REC "")...
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Successfully mounted /run/systemd/inaccessible/dir to /run/systemd/unit-root/run/credentials
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/run/systemd/incoming
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Followed source symlinks /run/systemd/propagate/caddy.service → /run/systemd/propagate/caddy.service.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Bind-mounting /run/systemd/propagate/caddy.service on /run/systemd/unit-root/run/systemd/incoming (MS_BIND "")...
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Successfully mounted /run/systemd/propagate/caddy.service to /run/systemd/unit-root/run/systemd/incoming
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/tmp
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Bind-mounting /tmp/systemd-private-f69532597e214c06939989bffc9584c4-caddy.service-GQU48b/tmp on /run/systemd/unit-root/tmp (MS_BIND|MS_REC "")...
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Successfully mounted /tmp/systemd-private-f69532597e214c06939989bffc9584c4-caddy.service-GQU48b/tmp to /run/systemd/unit-root/tmp
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/usr
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Bind-mounting /run/systemd/unit-root/usr on /run/systemd/unit-root/usr (MS_BIND|MS_REC "")...
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Successfully mounted /run/systemd/unit-root/usr to /run/systemd/unit-root/usr
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Applying namespace mount on /run/systemd/unit-root/var/tmp
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Bind-mounting /var/tmp/systemd-private-f69532597e214c06939989bffc9584c4-caddy.service-DIGXdO/tmp on /run/systemd/unit-root/var/tmp (MS_BIND|MS_REC "")...
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Successfully mounted /var/tmp/systemd-private-f69532597e214c06939989bffc9584c4-caddy.service-DIGXdO/tmp to /run/systemd/unit-root/var/tmp
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Remounted /run/systemd/unit-root/boot/efi.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Remounted /run/systemd/unit-root/boot.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Remounted /run/systemd/unit-root/etc.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Remounted /run/systemd/unit-root/run/credentials.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Remounted /run/systemd/unit-root/run/systemd/incoming.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Remounted /run/systemd/unit-root/usr.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: Remounted /run/systemd/unit-root/run/credentials.
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: caddy.service: Failed to locate executable /usr/bin/caddy: Permission denied
Sep 25 19:17:12 localhost.localdomain (caddy)[3169]: caddy.service: Failed at step EXEC spawning /usr/bin/caddy: Permission denied
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Child 3169 belongs to caddy.service.
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Main process exited, code=exited, status=203/EXEC
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Failed with result 'exit-code'.
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Service will not restart (restart setting)
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Changed start -> failed
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Job 779 caddy.service/start finished, result=failed
Sep 25 19:17:12 localhost.localdomain systemd[1]: Failed to start Caddy.
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Unit entered failed state.
Sep 25 19:17:12 localhost.localdomain systemd[1]: caddy.service: Releasing resources...

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Manually installation with reference to the official manual installation documentation and running as a systemd service.

a. System environment:

OS=openEuler 22.03 LTS-SP3 (treat it as CentOS-7 with more up-to-date kernel)
Architecture=amd x86
$ systemctl --version=systemd 249 (v249-80.oe2203sp3)

b. Command:

[user@localhost ~]$ sudo systemctl restart caddy
Job for caddy.service failed because the control process exited with error code.
See "systemctl status caddy.service" and "journalctl -xeu caddy.service" for details.

c. Service/unit/compose file:

# /etc/systemd/system/caddy.service
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

# /etc/caddy/Caddyfile
{
        auto_https disable_redirects
}

:8000 {
        reverse_proxy localhost:3000

        tls "/etc/caddy/cert/server_cert.crt" "/etc/caddy/cert/server_key.key" {
                key_type rsa4096
        }

        log {
                format console
                output file /var/log/caddy.log {
                        roll_size 100MiB
                        roll_local_time
                        roll_keep 10
                        roll_keep_for 30d
                }
        }
}

5. Links to relevant resources:

This post seems related but it doesn’t help.

Caddy permission denied update binary

1. The problem I’m having: i cant run my updated binary. I was running caddy binary with systemd service on ubuntu 22.04 on oracle free tier vps. I had this working before but wanted to update to the latest caddy version. So I used the download page and selected amd64 linux added my plugins and downloaded it to downloads. I Then sudo chown root:root caddy sudo mv caddy /usr/bin 2. Error messages and/or full log output: Jun 15 23:41:08 ubuntu-22-04 systemd[1]: Starting Caddy... ░░ Subject: A…

[henryfung3a27]

OK, I solved it by writing

ExecStart=sh -c "/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile"
ExecReload=sh -c "/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force"

in the caddy.service file. Somehow the sh -c trick works.

[francislavoie]

That’s really weird. Do you have something like SELinux going on? It might be blocking access to the binary to systemd.

[henryfung3a27]

Okay! Seems like SELinux is doing its thing.

$ sudo ausearch -m avc -f caddy -i
type=AVC msg=audit(09/25/2024 19:17:12.848:203) : avc:  denied  { execute } for  pid=3169 comm=(caddy) name=caddy dev="dm-0" ino=20064656 scontext=system_u:system_r:init_t:s0 tcontext=unconfined_u:object_r:user_home_t:s0 tclass=file permissive=0 

I will leave it as is for now and use the sh -c method. Thanks.

[francislavoie]

You might want to consider using systemd overrides instead of editing the existing service file (so that if we push changes to the service file, your changes don’t get overwritten).

Keep Caddy Running - Caddy Documentation