Well, that was the kick in the behind I needed to keep going for a bit longer, though I do have to say I’ve been out of my comfort zone for a while now. After trying a whole bunch of stuff that led nowhere, I might have made a little progress now. It turns out there are two issues I was grappling with:
- The trust store issue
- The ‘real’ root.crt location.
It turns out the second issue had a bearing on the first.
root.crt location
I was beating my head against the wall installing the root cert under /var/lib/caddy… into the trust store and not making any headway with mTLS.
The clue here was this note from @Rob789’s wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts :
NOTE: The absolute path for the certificate depends on how you run Caddy. Typically when you use systemd the absolute path will be
/var/lib/caddy/.local/share/caddy/pki/authorities/localbut if you run caddy withcaddy startthe path.local/share/caddy/pki/authorities/local
@Rob789 I’m curious to know where you came across this gem? Had I heeded the advice, my head might not have been as battered and bloodied 
This is what made me change tack. When I looked more closely at both locations this is what I saw:
root@caddy:~ # ls -l /.local/share/caddy/pki/authorities/local/
total 26
-rw------- 1 root wheel 676 May 19 12:27 intermediate.crt
-rw------- 1 root wheel 227 May 19 12:27 intermediate.key
-rw------- 1 root wheel 627 May 2 13:35 root.crt
-rw------- 1 root wheel 227 May 2 13:35 root.key
root@caddy:~ # ls -l /var/lib/caddy/.local/share/caddy/pki/authorities/local/
total 26
-rw------- 1 root wheel 676 May 10 11:06 intermediate.crt
-rw------- 1 root wheel 227 May 10 11:06 intermediate.key
-rw------- 1 root wheel 627 May 10 11:06 root.crt
-rw------- 1 root wheel 227 May 10 11:06 root.key
There was a more recent date for the intermediate cert in the former, which led me to suspect that I needed to install the root cert from that directory in the trust store. As caddy start is used to run caddy in the environment, based on the note, this reinforced which root.crt I needed to install in the trust store. Once I understood this, I began to make some progress. mTLS is still not working completely though, but I’ll get to that shortly.
Trust root CA cert
So, I found a couple of resources that seemed to be pertinent.
- How to install a private CA certificate on FreeBSD, and more recently…
- Howto install a private CA certificate? - a FreeBSD forum thread.
Combining elements of both, I think I have installed the correct cert in the system trust. Here’s a log of the steps I took:
root@caddy:~ # freebsd-version
12.2-RELEASE-p6
root@caddy:~ # cp /.local/share/caddy/pki/authorities/local/root.crt /etc/ssl/certs
root@caddy:~ # openssl rehash /etc/ssl/certs
root@caddy:~ # openssl s_client -connect acme.lan:443 | grep -i -e verify
depth=1 CN = Caddy Local Authority - ECC Intermediate
verify return:1
depth=0
verify return:1
Verify return code: 0 (ok)
Verify return code: 0 (ok)
root@caddy:~ #
Restarting the frontend Caddy service and checking the console log, there’s still the trust store error, but I assume that’s to be expected.
root@caddy:~ # tail --lines=20 /var/log/caddy.log
{"level":"info","ts":"2021-05-19T16:15:55.047+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00033a7e0"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412155.1001155,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412155.1057749,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000294690"}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-19T16:16:01.484+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/19 16:16:01 Note: NSS support is not available on your platform
2021/05/19 16:16:01 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-19T16:16:01.485+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","readymcgetty.com.au","xenografix.com.au","www.caffigoalkeeping.com","acme.lan","www.udance.com.au","udance.com.au","www.caffigoalkeeping.com.au","caffigoalkeeping.com.au","*.udance.com.au","caffigoalkeeping.com","www.readymcgetty.com.au"]}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-19T16:16:01.495+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=53177) - Caddy is running in the background
{"level":"info","ts":"2021-05-19T16:16:01.496+0800","logger":"tls","msg":"finished cleaning storage units"}
Restarting the backend Caddy service and checking the log, mTLS appears to be stirring into life!
{"level":"info","ts":"2021-05-19T16:22:28.408+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-19T16:22:28.408+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-19T16:22:28.411+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002e3110"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412548.539171,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412548.543643,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ec690"}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-19T16:22:28.550+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-19T16:22:28.572+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=28161) - Caddy is running in the background
Attempting to access test.udance.com.au, I see a 502 error in the browser, nothing in the backend Caddy log, but this error in the frontend log.
{"level":"error","ts":"2021-05-19T16:44:22.869+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"10.1.1.222:49381","proto":"HTTP/2.0","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wfwaf-authcookie-ce4a5970b2fd99774e0154e5e6b9716d=1%7Cadministrator%7Cd4b9b9e4e28bbee1e82f1b7f2b053da5fb2e984b668ed5b81d82bebf717d7436"],"Sec-Gpc":["1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005157315,"status":502,"err_id":"x10a03jus","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}
Have I installed the correct root cert in the system trust? I’m still not entirely sure. Can we get past this next stumbling block? That remains to be seen.
Relevant sections of the frontend Caddyfile…
...
# ACME server
acme.lan {
acme_server
tls internal
}
...
*.udance.com.au {
...
map {labels.3} {backend} {online} {mtls} {phpmyadmin} {
# HOSTNAME BACKEND ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
...
test test.lan:443 yes yes yes # test.udance.com.au
...
route {
...
# Secure backend communication
@mtls expression `{mtls} == "yes"`
reverse_proxy @mtls {backend} {
header_up Host {http.reverse_proxy.upstream.hostport}
header_up X-Forwarded-Host {host}
transport http {
tls
}
}
...
}
Relevant sections of the backend Caddyfile (full copy in post #29) …
{
...
acme_ca https://acme.lan/acme/local/directory
acme_ca_root /etc/ssl/certs/root.crt
}
test.lan {
...
}
Oh, once I worked out which root.crt to use, this appeared earlier in the day in the backend Caddy log. Seems promising.
{"level":"info","ts":"2021-05-19T14:39:19.704+0800","logger":"tls.renew","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:19.707+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"test.lan","remaining":-736110.707918771}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:20.069+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-19T14:39:20.138+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:44929","distributed":false}
{"level":"info","ts":"2021-05-19T14:39:20.492+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/BpA9sI3WDWa6fMtGGZufr5pRjKRKwo2l"}
{"level":"info","ts":"2021-05-19T14:39:20.704+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/gxvym5UeCVGotOR1rKJymjEj6AmDCq5X"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"releasing lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.706+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["test.lan"]}
{"level":"warn","ts":"2021-05-19T14:39:20.707+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T14:39:20.707+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["test.lan"],"new_expiration":"2021-05-19T18:39:20.000Z"}