# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# caddy run command or use the caddy-api.service file instead.
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddyfile or JSON config:
Our caddyfile configuration does not modify TLS configuration, its using defaults for local CA at this point, for this discussion its not necessary. If the discussion determines its needed I'll add it.
3. The problem I’m having:
I am attempting to use configure caddy to run as a service as a non root user (caddy) with a no login shell. I am slowly coming to the realization that there is no way to establish “trust” for the local CA from within caddy setup processes, if caddy is not running as root.
I’ve seen the other discussions on “why worry” regarding running caddy as root. This looks like the actual reason “why you must run caddy as root if you expect local auto-tls to work”.
Before completely redoing our system build I’m firing off a flare to the community on “has anyone gotten this working as non root user for caddy” before surrendering and switching our approach.
4. Error messages and/or full log output:
5. What I already tried:
6. Links to relevant resources:
Why we are chasing non root services configuration:
You can run the internal issuer/local CA as non-root, but then Caddy won’t be able to install the root CA cert automatically in your system’s trust store. But you can do that yourself manually.
You can find the root CA cert in Caddy’s storage location (depends how you run Caddy), at pki/authorities/local/root.crt
You can find plenty of guides online for how to install certificates in linux or your browsers.
Francis, thanks. Indeed I do know how to do that manually. We will look at first setup being the point of manual insertion & configuration.
When I first posted I was lamenting CA rollover, however, when examining the Caddy root CA, its a 10 year lifetime (I was assuming its 1 year), so worrying about automation to capture the CA rollover is moot at that scale.
root@elite-pi:/home/caddy/.local/share/caddy/pki/authorities/local# openssl x509 -in /home/caddy/.local/share/caddy/pki/authorities/local/root.crt -noout -text
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
fc:31:3e:dc:28:55:8f:51:86:8d:97:47:56:11:1a:fb
Signature Algorithm: ecdsa-with-SHA256
Issuer: CN = Caddy Local Authority - 2021 ECC Root
Validity
Not Before: Mar 24 04:06:13 2021 GMT
Not After : Jan 31 04:06:13 2031 GMT
Subject: CN = Caddy Local Authority - 2021 ECC Root
One other thing I noticed was the announce over sudo being available for the caddy user, which means its at least trying to get it installed as non root user.
We could provide a restrictive sudoers file definition for caddy to execute this step… is there a full breakdown of the commands caddy is attempting to run (beyond the “tee” indicated as I assume there is a chain of errors behind it waiting) I’ve been looking at caddy/autohttps.go at master · caddyserver/caddy · GitHub but I’m not seeing when/where the system level commands to actually inject the caddy CA are being run to create a set of more specific sudoers file rules for what caddy can do as root…
Mar 23 22:06:14 elite-pi caddy[8649]: {"level":"warn","ts":1616558774.066763,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Mar 23 22:06:14 elite-pi caddy[8649]: 2021/03/23 22:06:14 not NSS security databases found
Mar 23 22:06:14 elite-pi caddy[8649]: 2021/03/23 22:06:14 define JAVA_HOME environment variable to use the Java trust
Mar 23 22:06:14 elite-pi sudo[8659]: pam_unix(sudo:auth): conversation failed
Mar 23 22:06:14 elite-pi sudo[8659]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Mar 23 22:06:14 elite-pi sudo[8659]: caddy : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_335221152435941021957087867284048911099.crt
Mar 23 22:06:14 elite-pi caddy[8649]: {"level":"error","ts":1616558774.0844543,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
That “tee” command being referenced is probably part of a larger string of commands piping the output of that certificate into another larger set of commands?
So blasting sudoers to allow caddy to do anything works (as expected) but it would be nice to define a “template” to share with community on what the actual commands are to exception for caddy in the /etc/sudoers file.
A cookie crumb trail into github on where to be reading through to understand the system commands would be Awesome </jazz hands>
So its looking like the “tee” that is visible in the logging, is the OS level command being run that needs priveledged access.
The next command with sudo is here, and its responsible for establishing the system trust once tee has placed the caddy local CA root certificate in place to be configured over. This is what invokes the proper command, but the the logic that selects the command based on the implementation is in the if/then logic’s init function…
The init function code is checking if paths exist based on the linux distro its on, and then picking out its proper command to use in the systemTrustCommand. That logic is laid out here:
So based on our build environment (raspbian lite / debian on ARM7) the command line would be
update-ca-certificates
SO our /etc/sudoers will have the following (if you are playing along from home folks, use sudo visudo !!!)
This still needs fine tuning to strip away caddy being able to run commands as ALL users, I’m reading for more context there and will update…
Ok so we are still missing a step in that the final trust configuration is still failing with just those two commands enabled for cady.
Mar 30 09:02:32 elite-pi caddy[1275]: {"level":"error","ts":1617116552.1770575,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
I’ll continue to review, might be subcommands within the update-ca-certificates
Oh this is absolutely brutal, I give up. So the difference between what is working and what doesn’t work is the fact that a user with sudo that has a login shell for their account, performs the certificate update with no issue, while the hardened user we intend to use that has /usr/sbin/nologin fails
As the caddy user who has /usr/sbin/nologin but identical sudoers file config for all/all nopasswd, it fails.
caddy:x:998:1002:Caddy web server:/var/lib/caddy:/usr/sbin/nologin
Mar 30 19:33:41 elite-pi caddy[988]: {"level":"warn","ts":1617154421.0475287,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Mar 30 19:33:41 elite-pi caddy[988]: 2021/03/30 19:33:41 not NSS security databases found
Mar 30 19:33:41 elite-pi caddy[988]: 2021/03/30 19:33:41 define JAVA_HOME environment variable to use the Java trust
Mar 30 19:33:41 elite-pi sudo[998]: caddy : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2021_ECC_Root_295918027004045786159331586156969744716.crt
Mar 30 19:33:41 elite-pi sudo[998]: pam_unix(sudo:session): session opened for user root by (uid=0)
Mar 30 19:33:41 elite-pi sudo[998]: pam_unix(sudo:session): session closed for user root
Mar 30 19:33:41 elite-pi caddy[988]: {"level":"error","ts":1617154421.0718782,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
where as the user that has login shell works
eeuser:x:1001:1001::/home/eeuser:/bin/bash
2021/03/31 01:47:42.142 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}
2021/03/31 01:47:42.144 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0x3538230"}
2021/03/31 01:47:42.159 INFO http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/03/31 01:47:42.160 INFO http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2021/03/31 01:47:42.164 INFO http.handlers.auth_portal JWT token name found {"instance_name": "portal-1", "token_name": "access_token"}
2021/03/31 01:47:42.165 WARN http.handlers.auth_portal JWT token origin not found, using default {"instance_name": "portal-1"}
2021/03/31 01:47:42.166 INFO http.handlers.auth_portal local backend configuration {"db_path": "/opt/caddy/auth/user_db.json", "require_mfa": false}
2021/03/31 01:47:42.172 INFO http.handlers.auth_portal validating local backend {"db_path": "/opt/caddy/auth/user_db.json"}
2021/03/31 01:47:42.188 INFO http.handlers.auth_portal JWT token validator provisioned {"instance_name": "portal-1", "access_list": [{"action":"allow","values":["anonymous","guest","*"],"claim":"roles"}]}
2021/03/31 01:47:42.190 INFO http.handlers.auth_portal provisioned plugin instance {"instance_name": "portal-1", "started_at": "2021/03/31 01:47:42.164"}
2021/03/31 01:47:42.192 INFO http.authentication.providers.jwt provisioned plugin instance {"instance_name": "jwt-4", "started_at": "2021/03/31 01:47:42.191"}
2021/03/31 01:47:42.283 WARN pki.ca.local installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2021/03/30 19:47:42 not NSS security databases found
2021/03/30 19:47:42 define JAVA_HOME environment variable to use the Java trust
2021/03/30 19:47:46 certificate installed properly in linux trusts
2021/03/31 01:47:46.192 INFO tls cleaned up storage units
2021/03/31 01:47:46.193 INFO http enabling automatic TLS certificate management {"domains": ["192.168.37.127", "mg.int.elite-env.com"]}
So we are falling back to our original plan of running caddy trust in a controlled way during setup, now that the caddy user has shell we can do it from our setup script.
/bin/su -c "(/usr/bin/caddy trust)" - caddy
I dont have time to rebuild based on caddy being root to test, but I’m really wondering if caddy sets up trust from a caddyfile that is already configured with https and local_certs as setting, when starting up from a systemd unit file… If anyone could share journalctl -xe output of a root based caddy starting up from systemctl, properly setting up local CA in the current 2.4.0beta that would be very helpful.