Understanding accessible paths via systemd service on Arch Linux

Version: v2.5.2

As in Caddy File_Server's Files Give 404 Error after some trail-and-error I found out that I could neither run a file server from my home directory nor from /tmp (with all permissions and ownership granted) if I run the systemd service, but it works if launched manually.

So I looked into the service files:

❯ bat -P --style=header,rule /usr/lib/**/caddy.*
File: /usr/lib/systemd/system/caddy.service
# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network-online.target
Wants=network-online.target systemd-networkd-wait-online.service
StartLimitIntervalSec=14400
StartLimitBurst=10

[Service]
User=caddy
Group=caddy
Environment=XDG_DATA_HOME=/var/lib
Environment=XDG_CONFIG_HOME=/etc
ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
ExecStopPost=/usr/bin/rm -f /run/caddy/admin.socket

# Do not allow the process to be restarted in a tight loop. If the
# process fails to start, something critical needs to be fixed.
Restart=on-abnormal

# Use graceful shutdown with a reasonable timeout
TimeoutStopSec=5s

LimitNOFILE=1048576
LimitNPROC=512

# Hardening options
AmbientCapabilities=CAP_NET_BIND_SERVICE
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
DevicePolicy=closed
LockPersonality=true
MemoryAccounting=true
MemoryDenyWriteExecute=true
NoNewPrivileges=true
PrivateDevices=true
PrivateTmp=true
ProcSubset=pid
ProtectClock=true
ProtectControlGroups=true
ProtectHome=true
ProtectHostname=true
ProtectKernelLogs=true
ProtectKernelModules=true
ProtectKernelTunables=true
ProtectProc=invisible
ProtectSystem=strict
RemoveIPC=true
ReadWritePaths=/var/lib/caddy /var/log/caddy /run/caddy
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true

[Install]
WantedBy=multi-user.target
─────────────────────────────────────────────────────────────────────────
File: /usr/lib/sysusers.d/caddy.conf
u caddy - "caddy daemon" /var/lib/caddy
─────────────────────────────────────────────────────────────────────────
File: /usr/lib/tmpfiles.d/caddy.conf
d /var/lib/caddy 0750 caddy caddy
d /var/log/caddy 0750 caddy caddy
d /run/caddy 0750 caddy caddy

The paths here seem to deviate from the documentation: Keep Caddy Running β€” Caddy Documentation
But what confuses me even more is that I can put my files in /usr/share/caddy and it works, despite this path seemingly not being mentioned in the allowed paths. Is that a peculiarity of systemd, that this path is always allowed?
What about /var/www/html, which also works?
I am rather confused how the allowed paths are determined and would love to get some insight :slight_smile:

Finally, only halfway related:
Why does caddy serve an empty response on error by default? This is quite confusing, and as seen in the documentation it is a no-brainer to change. Caddy seems to aim for good defaults, and returning at least a simple error page seems a lot saner than a blank one.
It would be nice if it was possible to show or at least log if the file could not be accessed due to a permission error, I got a generic 404 and was quite confused for a while…

Interesting, that doesn’t look like the service file we provide. Maybe arch’s distro is doing something custom? I’ll ping a team member, maybe he can help explain.

We already do that, just check the log or the status code! (Enable access logging if you haven’t already. That’s just the log directive.)

Ye, the arch package’s systemd service leverages some additional systemd β€œhardening”, while the reference systemd service doesn’t:

vs


ProtectHome=true does that:

[…] If true, the directories /home/, /root, and /run/user are made inaccessible and empty for processes invoked by this unit.

and that’s due to PrivateTmp=true:

[…] If true, sets up a new file system namespace for the executed processes and mounts private /tmp/ and /var/tmp/ directories inside it that are not shared by processes outside of the namespace. This is useful to secure access to temporary files of the process, but makes sharing between processes via /tmp/ or /var/tmp/ impossible. If true, all temporary files created by a service in these directories will be removed after the service is stopped.


Caddy already returns a http/403 whenever it runs into a permission error while serving file_server :slight_smile:
Sure, will still be a blank page, which I get may seem odd coming to almost any other webserver.
But in your specific case (or the arch package’s version of caddy in general, actually), caddy has literally no chance of seeing those files in /tmp/ or /home/ - due to those additional systemd sandboxing flags.

Also, be aware that if you have a file in /home/xerus/website/index.html and chmod /home/xerus/website to be world-accessible, you will still be unable to access that folder (and any files within it) as any other user, because /home/xerus also needs to be world-accessible.

To reach a file in a tree of folders on Linux, the requesting user has to have permissions on any folder leading to that tree (e.g. /home, /home/xerus and /home/xerus/website) and those folders need to have the execution bit set.

See Reverse proxy + static file serving results in 403 (Forbidden) for static files - #2 by IndeedNotJames for a bit more details :innocent:

I would highly recommend considering /srv/ or /var/www/ instead :slight_smile:

3 Likes

Thanks for the detailed response!
Maybe we can find a spot in the documentation to add that permanently?

We do link to the package files from our install page:

https://archlinux.org/packages/community/x86_64/caddy/

And from there you can view the up-to-date, actual files. I’d rather not duplicate what an individual package distribution does (which can change any time) in our docs, and risk them getting out of date.

I would argue arch-specific things are best documented in Caddy - ArchWiki

Maybe some quick note that the arch package uses a non-standard service files would make sense though, I think.

This package comes with both of Caddy’s systemd service unit files, but does not enable them by default.

from Install β€” Caddy Documentation reads a bit misleading to me personally :thinking:

2 Likes