V2 Get it working with internal CA in systemd?

Help
1

1. My Caddy version (caddy version):

$ caddy version
v2.0.0-rc.3 h1:z2H/QnaRscip6aZJxwTbghu3zhC88Vo8l/K57WUce4Q=

2. How I run Caddy:

a. System environment:

$ uname -a
Linux 4.15.0-96-generic #97-Ubuntu SMP Wed Apr 1 03:25:46 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
$ systemd --version
systemd 237
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid

c. Service/unit/compose file:

[Unit]
Description=Radd Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
Restart=on-failure
ExecStart=/usr/bin/caddy run --config /etc/caddy/Raddfile --environ --adapter caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Raddfile --adapter caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
#ProtectHome=true // TODO try enabling this
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
admin localhost:2020
debug
}

https://192.168.196.249 {
    bind 192.168.196.249

    tls internal

   	log {
    	output file /var/log/caddy/ctrl_access.log
   		format single_field common_log
   	}

   	encode gzip zstd

   	handle /spi/* {
   		reverse_proxy localhost:8224
   	}

   	handle {
        root * /var/www/ctrl
   		try_files {path} /index.html
   		file_server
   	}
}

3. The problem I’m having:

I want to run Caddy on a private VPN IP with no exposure to the Internet. So I’m trying to use the internal CA for https. The only thing I could find in the docs is that it might ask for a password. Obviously this doesn’t work when running in systemd.

See errors below.

In case it’s not obvious, I’m trying to run 2 instances of caddy, I named the 2nd one radd so it’s obvious which one I"m looking at in a given moment.

4. Error messages and/or full log output:

5. What I already tried:

Just running it in systemd causes it to fail because it tries to run sudo but can’t:

Apr 16 22:56:34 caddy[32135]: caddy.HomeDir=/var/lib/caddy
Apr 16 22:56:34 caddy[32135]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Apr 16 22:56:34 caddy[32135]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Apr 16 22:56:34 caddy[32135]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Apr 16 22:56:34 caddy[32135]: runtime.GOOS=linux
Apr 16 22:56:34 caddy[32135]: runtime.GOARCH=amd64
Apr 16 22:56:34 caddy[32135]: runtime.Compiler=gc
Apr 16 22:56:34 caddy[32135]: runtime.NumCPU=1
Apr 16 22:56:34 caddy[32135]: runtime.GOMAXPROCS=1
Apr 16 22:56:34 caddy[32135]: runtime.Version=go1.14.2
Apr 16 22:56:34 caddy[32135]: os.Getwd=/
Apr 16 22:56:34 caddy[32135]: LANG=C
Apr 16 22:56:34 caddy[32135]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Apr 16 22:56:34 caddy[32135]: HOME=/var/lib/caddy
Apr 16 22:56:34 caddy[32135]: LOGNAME=caddy
Apr 16 22:56:34 caddy[32135]: USER=caddy
Apr 16 22:56:34 caddy[32135]: INVOCATION_ID=3831145c2ff74f1fb56ac2c9c56127ea
Apr 16 22:56:34 caddy[32135]: JOURNAL_STREAM=9:371889
Apr 16 22:56:34 caddy[32135]: {"level":"info","ts":1587077794.7355578,"msg":"using provided configuration","config_file":"/etc/caddy/Raddfile","config_adapter":"caddyfile"}
Apr 16 22:56:34 caddy[32135]: {"level":"info","ts":1587077794.7375135,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2020","enforce_origin":false,"origins":["localhost:2020","[::1]:2020","127.0.0.1:2020"]}
Apr 16 22:56:34 caddy[32135]: 2020/04/16 22:56:34 [INFO][cache:0xc0006b0d20] Started certificate maintenance routine
Apr 16 22:56:34 caddy[32135]: {"level":"info","ts":1587077794.7463655,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 16 22:56:34 caddy[32135]: {"level":"warn","ts":1587077794.7868187,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Apr 16 22:56:34 caddy[32135]: 2020/04/16 22:56:34 not NSS security databases found
Apr 16 22:56:34 caddy[32135]: 2020/04/16 22:56:34 define JAVA_HOME environment variable to use the Java trust
Apr 16 22:56:34 sudo[32140]: pam_unix(sudo:auth): conversation failed
Apr 16 22:56:34 sudo[32140]: pam_unix(sudo:auth): auth could not identify password for [caddy]
Apr 16 22:56:34 sudo[32140]:    caddy : user NOT in sudoers ; TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2020_ECC_Root_202450710866994073558541654554824492202.crt
Apr 16 22:56:34 caddy[32135]: {"level":"error","ts":1587077794.8941307,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Apr 16 22:56:34 caddy[32135]: {"level":"info","ts":1587077794.8949153,"logger":"tls","msg":"cleaned up storage units"}
Apr 16 22:56:34 caddy[32135]: {"level":"debug","ts":1587077794.896629,"logger":"http","msg":"starting server loop","address":"192.168.196.249:443","http3":false,"tls":true}
Apr 16 22:56:34 caddy[32135]: {"level":"debug","ts":1587077794.8967886,"logger":"http","msg":"starting server loop","address":"192.168.196.249:80","http3":false,"tls":false}
Apr 16 22:56:34 caddy[32135]: {"level":"info","ts":1587077794.896923,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.196.249"]}
Apr 16 22:56:34 caddy[32135]: 2020/04/16 22:56:34 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.196.249]: no OCSP server specified in certificate
Apr 16 22:56:34 caddy[32135]: {"level":"info","ts":1587077794.8977227,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Apr 16 22:56:34 caddy[32135]: {"level":"info","ts":1587077794.8978462,"msg":"serving initial configuration"}
Apr 16 22:56:43 systemd[1]: Stopping Radd Web Server...
Apr 16 22:56:43 caddy[32135]: {"level":"info","ts":1587077803.2686393,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
Apr 16 22:56:43 caddy[32135]: 2020/04/16 22:56:43 [INFO][cache:0xc0006b0d20] Stopped certificate maintenance routine
Apr 16 22:56:43 caddy[32135]: {"level":"info","ts":1587077803.268778,"msg":"shutdown done","signal":"SIGTERM"}
Apr 16 22:56:43 systemd[1]: Stopped Radd Web Server.

So I (temporarily) added the caddy user to sudo group. The sudo problem disappears but it still fails with an error. It does run, but any attempt to access it from a browser gets the security warning. But something is failing even with sudo privs:

Apr 16 22:56:44 systemd[1]: Started Radd Web Server.
Apr 16 22:56:44 caddy[32245]: caddy.HomeDir=/var/lib/caddy
Apr 16 22:56:44 caddy[32245]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Apr 16 22:56:44 caddy[32245]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Apr 16 22:56:44 caddy[32245]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Apr 16 22:56:44 caddy[32245]: runtime.GOOS=linux
Apr 16 22:56:44 caddy[32245]: runtime.GOARCH=amd64
Apr 16 22:56:44 caddy[32245]: runtime.Compiler=gc
Apr 16 22:56:44 caddy[32245]: runtime.NumCPU=1
Apr 16 22:56:44 caddy[32245]: runtime.GOMAXPROCS=1
Apr 16 22:56:44 caddy[32245]: runtime.Version=go1.14.2
Apr 16 22:56:44 caddy[32245]: os.Getwd=/
Apr 16 22:56:44 caddy[32245]: LANG=C
Apr 16 22:56:44 caddy[32245]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Apr 16 22:56:44 caddy[32245]: HOME=/var/lib/caddy
Apr 16 22:56:44 caddy[32245]: LOGNAME=caddy
Apr 16 22:56:44 caddy[32245]: USER=caddy
Apr 16 22:56:44 caddy[32245]: INVOCATION_ID=4f9dfe99d2da4260b060ce8731eaba41
Apr 16 22:56:44 caddy[32245]: JOURNAL_STREAM=9:372913
Apr 16 22:56:44 caddy[32245]: {"level":"info","ts":1587077804.6654708,"msg":"using provided configuration","config_file":"/etc/caddy/Raddfile","config_adapter":"caddyfile"}
Apr 16 22:56:44 caddy[32245]: {"level":"info","ts":1587077804.6675344,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2020","enforce_origin":false,"origins":["localhost:2020","[::1]:2020","127.0.0.1:2020"]}
Apr 16 22:56:44 caddy[32245]: {"level":"info","ts":1587077804.6759996,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Apr 16 22:56:44 caddy[32245]: 2020/04/16 22:56:44 [INFO][cache:0xc0006b0d20] Started certificate maintenance routine
Apr 16 22:56:44 caddy[32245]: {"level":"warn","ts":1587077804.7195303,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
Apr 16 22:56:44 caddy[32245]: 2020/04/16 22:56:44 not NSS security databases found
Apr 16 22:56:44 caddy[32245]: 2020/04/16 22:56:44 define JAVA_HOME environment variable to use the Java trust
Apr 16 22:56:44 sudo[32252]:    caddy : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/bin/tee /usr/local/share/ca-certificates/Caddy_Local_Authority_-_2020_ECC_Root_202450710866994073558541654554824492202.crt
Apr 16 22:56:44 sudo[32252]: pam_unix(sudo:session): session opened for user root by (uid=0)
Apr 16 22:56:44 sudo[32252]: pam_unix(sudo:session): session closed for user root
Apr 16 22:56:44 caddy[32245]: {"level":"error","ts":1587077804.8249412,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute sudo: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
Apr 16 22:56:44 caddy[32245]: {"level":"info","ts":1587077804.8258626,"logger":"tls","msg":"cleaned up storage units"}
Apr 16 22:56:44 caddy[32245]: {"level":"debug","ts":1587077804.8260424,"logger":"http","msg":"starting server loop","address":"192.168.196.249:443","http3":false,"tls":true}
Apr 16 22:56:44 caddy[32245]: {"level":"debug","ts":1587077804.8262112,"logger":"http","msg":"starting server loop","address":"192.168.196.249:80","http3":false,"tls":false}
Apr 16 22:56:44 caddy[32245]: {"level":"info","ts":1587077804.8263278,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["192.168.196.249"]}
Apr 16 22:56:44 caddy[32245]: 2020/04/16 22:56:44 [WARNING] Stapling OCSP: no OCSP stapling for [192.168.196.249]: no OCSP server specified in certificate
Apr 16 22:56:44 caddy[32245]: {"level":"info","ts":1587077804.8272402,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Apr 16 22:56:44 caddy[32245]: {"level":"info","ts":1587077804.8273797,"msg":"serving initial configuration"}

So can someone tell me how to get the internal CA working while running in systemd? Thanks.

6. Links to relevant resources:

2

You should be able to find the root CA certificate at /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt, i.e. the value of the caddy.AppDataDir log line, plus the "certificate_file" value found in one of the log messages.

You can manually install this in your trust store. You can do this manually by following these instructions: How do I install a root certificate? - Ask Ubuntu

3

Which browser are you using? Some browsers like Firefox need to be restarted before new trust settings take effect.

Also, and this should be obvious, but the cert will only be trusted on the local machine by default; if you are accessing this site remotely, you’ll need to install Caddy’s root cert into your own local machine’s trust store, as Francis suggested.

4

Ok. Thanks to both.

I’m way out of my depth on this topic. Obviously using an internal CA isn’t as simple as I imagined - requiring synchronized changes on both server and client.

I may have to try to find some less complex solution.

Thanks again.

5

Installing the root CA certificate is only a one-time thing (per client).

6

The internal CA is meant for local dev sites, not production deployments, per-se.

2.1 will have an embedded ACME server that will make it easier, but you still have to trust the server’s CA certificate. Maybe a later version will automate that, too. (Won’t be particularly hard but you’d still need elevated privileges for this to work.)

And as Francis said, it’s just a one-time thing.

7

Also, why is this needed? Usually this is an anti-pattern or a yellow flag. I wonder if we can simplify your setup and get you down to 1 instance.

8

2 instances:

Yes. Totally agree.

The end design is 1 caddy, 2 sites. I have the “main” one running alpha production and I didn’t want to disturb it (as it does have actual users on it) while I’m trial-and-error hacking my way thru getting this second site working with https on internal CA.

I’ll fold them together shortly.

Thanks for pointing that out.

1 Like
9

Gotcha, thanks for explaining. Keep us posted!

10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.