How to run Caddy as a daemon - permission denied

I have a Caddy setup that works fine when I run it from the terminal, but now I am trying to start it with systemd. This results in the following error from the systemd caddy service log:

Mar 23 16:45:34 patchwork-starfish caddy[122554]: Error: reading config from file: open /etc/caddy/Caddyfile: permission denied

Caddy’s user does have permission to read it:

sudo -u caddy cat /etc/caddy/Caddyfile
# Prints out Caddyfile as expected

In my setup /etc/caddy/Caddyfile is a symlink. I tried removing the link and copying the file over instead, resulting in this new error:

Mar 23 16:38:24 patchwork-starfish caddy[121585]: Error: loading http app module: provision http: loading pki app module: provision pki: provisioning CA 'local': loading root cert: open /home/den-antares/.local/share/caddy/pki/authorities/local/root.crt: permission denied

I’ve tried granting the Caddy user access to those directories, editing /usr/lib/systemd/system/caddy.service to comment out ProtectHome and ProtectSystem, and setting the unit file to use my main user. No effect.

Eventually I tried disabling SELinux. Then Caddy was able to start, but attempting to visit the server (at https://localhost:8443/) resulted in the following error page:


Secure Connection Failed

An error occurred during a connection to localhost:8443. Peer’s certificate has an invalid signature.

...

At this point I’m pretty much completely lost. Is there a guide somewhere to all the file permissions fixes to run Caddy as a systemd daemon? I’m trying to figure out:

  • How do I get Caddy to follow symlinks? (So the Caddyfile can be a symlink to my repo and guarantee that it doesn’t get out of sync with the repo version.)
  • Can I run Caddy with SELinux enabled? I’d prefer to, but it’s not a strict requirement.
  • How do I get HTTPS to work when Caddy is running from systemd instead of from the temrinal?

If you cannot output the Caddyfile without using sudo, then the user running Caddy is unlikely to have the correct permissions. If Caddy is running as a systemd .service under the user patchwork-starfish, then either /etc/caddy/Caddyfile needs to be owned by that user, be a part of a group that can at least read the file, or have all others outside the user:group be able to read/execute. That would be chmod 775.

I would recommend that you sudo chown the Caddyfile to have the user own it, or preferrably, change the directory that Caddy reads a Caddyfile from. I believe it SHOULD be reading from $HOME/.config/caddy, so it should assume that a Caddyfile is in that directory. If it isn’t, then edit the caddy.service file.

ExecStart=/usr/bin/caddy run --environ --config %h/.config/caddy/Caddyfile

If I’m wrong, someone please correct me.

They’re using sudo -u caddy, so they’re switching to user caddy and executing the cat command as caddy user. So, the permissions for the user caddy to access the file seem to be OK.

I believe patchwork-starfish is a server name rather than a user name.

I don’t see why it should not run with SELinux enabled. You may need to make some adjustments, but it should run just fine. Can you run:

sudo audit2allow -a

In case you’re not familiar with SELinux, check, for example, this page.

Without seeing your Caddyfile, we’re pretty much too :smiley: But seriously, if you could show us:

  • your Caddyfile
  • ls -slaZ /etc/caddy/
  • sudo audit2allow -a

we might be able to help a bit more.

2 Likes

Here’s my Caddyfile:

{
	# Not necessary, I just don't like the admin API
	admin off
	
	http_port 8080
	
	# Required for Quest compatibility. If OCSP stapling is left on, serving files to
	# Quest may work for up to one week, but will then fail due to expiration issues.
	# The OCSP bug can be confirmed by leaving OCSP stapling on, waiting 8 days, and
	# then running `curl -v --cert-status https://HOSTNAME/``, which will end with
	# the error `curl: (91) OCSP response has expired`.
	ocsp_stapling off
}

localhost:8443 {
	file_server browse {
		root /www-tir-na-nog/root
	}

	reverse_proxy /api/* localhost:9000

	forward_auth /files/restricted/* localhost:9000 {
		uri /verify
	}

	log {
		output file /www-tir-na-nog/access.log
	}
}

Note: The hostname has been temporarily set to localhost for testing.

My caddy.service file:

# caddy.service
#
# For using Caddy with a config file.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy web server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
Type=notify
User=caddy
Group=caddy
#User=den-antares
#Group=den-antares
#ExecStartPre=/usr/bin/caddy validate --config /etc/caddy/Caddyfile
#ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
#ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
ExecStartPre=/usr/bin/caddy validate --config /www-tir-na-nog/Caddyfile
ExecStart=/usr/bin/caddy run --environ --config /www-tir-na-nog/Caddyfile
ExecReload=/usr/bin/caddy reload --config /www-tir-na-nog/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
#PrivateTmp=true
#ProtectHome=true
ProtectHome=false
#ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

When I run caddy from systemd with User=caddy, I get the follow error:

den-antares@patchwork-starfish www-tir-na-nog$ curl https://localhost:8443/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

When I run with User=den-antares, HTTPS does work. I’d prefer to run it under the caddy user if possible.

Audit2allow shows a lot of stuff:

#============= httpd_t ==============
allow httpd_t config_home_t:dir { add_name create write };
allow httpd_t config_home_t:file { create open write };

#!!!! This avc can be allowed using the boolean 'httpd_read_user_content'
allow httpd_t config_home_t:file open;
allow httpd_t data_home_t:dir { add_name create remove_name write };

#!!!! This avc can be allowed using the boolean 'httpd_read_user_content'
allow httpd_t data_home_t:file { getattr open read };
allow httpd_t data_home_t:file { create open unlink write };
allow httpd_t default_t:dir { add_name read };
allow httpd_t default_t:file { append create getattr open read write };
allow httpd_t default_t:lnk_file { getattr read };
allow httpd_t fprintd_t:dbus send_msg;

#!!!! This avc can be allowed using the boolean 'httpd_read_user_content'
allow httpd_t home_cert_t:file { getattr open read };
allow httpd_t home_cert_t:file write;
allow httpd_t pam_var_run_t:dir read;
allow httpd_t pam_var_run_t:file { lock open read write };

#!!!! This avc can be allowed using the boolean 'daemons_dump_core'
allow httpd_t root_t:file getattr;

#!!!! This avc can be allowed using the boolean 'httpd_mod_auth_pam'
allow httpd_t self:capability audit_write;

#!!!! This avc can be allowed using the boolean 'httpd_mod_auth_pam'
allow httpd_t self:netlink_audit_socket nlmsg_relay;
allow httpd_t shadow_t:file { getattr open read };
allow httpd_t sudo_db_t:dir read;

#!!!! This avc can be allowed using the boolean 'nis_enabled'
allow httpd_t unreserved_port_t:udp_socket name_bind;

#!!!! This avc can be allowed using the boolean 'httpd_read_user_content'
allow httpd_t user_home_t:dir read;
allow httpd_t user_home_t:dir add_name;

#!!!! This avc can be allowed using the boolean 'httpd_read_user_content'
allow httpd_t user_home_t:file { open read };
allow httpd_t user_home_t:file { create open write };

#!!!! This avc can be allowed using the boolean 'httpd_enable_homedirs'
allow httpd_t user_home_t:lnk_file read;

#============= iiosensorproxy_t ==============

#!!!! This avc is allowed in the current policy
allow iiosensorproxy_t self:unix_dgram_socket create;
allow iiosensorproxy_t sysfs_t:dir write;
allow iiosensorproxy_t sysfs_t:file write;
allow iiosensorproxy_t syslogd_var_run_t:dir search;

#============= systemd_homed_t ==============
allow systemd_homed_t var_t:dir read;

#============= virtqemud_t ==============
allow virtqemud_t user_home_dir_t:file { relabelfrom relabelto setattr write };

…but in my experience, trying to fix SELinux issues with audit2allow is a fool’s errand. Sure, it’ll work at first, but then your server will crash later at the most inconvenient possible time and there will be a new permission that needs to be added. Then again. And again. And again. Basically if you have to edit SELinux stuff you are guaranteed months of instability and debugging. If it comes down to it, disabling SELinux is generally the better option.

“You might be holding it wrong,” I’d say /jk :slight_smile:

I used to write my own SELinux modules before switching to containers. The best way to handle it is to run your app with SELinux enabled but in permissive mode rather than enforcing. Let it run this way for a week or two (often less) while SELinux logs all policy violations. Then, based on those logs, you write your SELinux module.

After that, monitor it again in permissive mode for about a week. Once everything looks good, you can switch to enforcing mode. I’ve never had issues with SELinux using this approach.

That said, a bunch of your httpd_t issues can be fixed with a few simple commands. audit2allow is telling you exactly what to do:

setsebool -P httpd_read_user_content 1
setsebool -P daemons_dump_core 1
setsebool -P httpd_mod_auth_pam 1
setsebool -P nis_enabled 1
setsebool -P httpd_enable_homedirs 1

Hope that helps!

Also, one option is to relabel the files and directories Caddy is using in your case to match the expected SELinux context for web content.

2 Likes

That might be worth trying, though I doubt I’ll have time to spare on messing with SELinux any time soon.

I’m slowly (very slowly) working out the (numerous) issues with getting HTTPS working under the caddy user:

# Disable SELinux
setenforce 0

# Create certificates
openssl req -x509 -out test-cert.pem -keyout test-key.pem \
	-newkey rsa:3072 -nodes -sha256 -subj "/CN=$$HOSTNAME" \
	-days 10000
chown caddy:caddy test-cert.pem test-key.pem

Then add tls /path/to/test-cert.pem /path/to/test-key.pem to the relevant host in your Caddyfile.

Then add skip_install_trust to the general config section of your Caddyfile.

What I think is going on is:

  1. Fedora’s Caddy package borked up the SELinux stuff.
  2. When Caddy starts it attempts to set up an internal CA, which requires root access. This must be disabled to allow caddy to run as a non-sudo user. (I found some Github threads that suggested a caddy trust command could also be used, but I couldn’t get that to work).
  3. Because the internal CA has to be disabled, certificates must be supplied from elsewhere.

While the automatic HTTPS is nice for setting up a production web site on a public server, setting up a test server that isn’t on a public IP with a registered domain name is easily the most difficult HTTPS setup I’ve done.