mTLS under FreeBSD

This made me realise I’d copied over the incorrect certificate to the backend Caddy server. I had copied across /.local/share/caddy/pki/authorities/local/root.crt instead of /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt.

@Rob789 The above paragraph in your wiki article under the heading Backends should be updated to use the absolute path to avoid any ambiguity:

root@caddy:~ # sudo HOME=/var/lib/caddy caddy trust
2021/05/10 13:40:41.376 WARN    ca.local        installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2021/05/10 21:40:41 Note: NSS support is not available on your platform
2021/05/10 21:40:41 define JAVA_HOME environment variable to use the Java trust
trust: trust not supported

After copying across the correct cert to the backend server, I then restarted both frontend and backend Caddy services and attempted to access the test site. Sadly still no go :cry:

Frontend caddy log extract

{"level":"info","ts":"2021-05-10T21:41:44.437+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:44.437+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:44.443+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00035a690"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654104.5705228,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654104.6052015,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:44.613+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-10T21:41:44.614+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00032ff80"}
{"level":"info","ts":"2021-05-10T21:41:44.644+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:44.645+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-10T21:41:57.225+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/10 21:41:57 Note: NSS support is not available on your platform
2021/05/10 21:41:57 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-10T21:41:57.226+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","readymcgetty.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","udance.com.au","www.caffigoalkeeping.com.au","www.readymcgetty.com.au","xenografix.com.au","acme.lan"]}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-10T21:41:57.258+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=96255) - Caddy is running in the background
{"level":"info","ts":"2021-05-10T21:41:57.260+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"error","ts":"2021-05-10T21:42:08.298+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.166.143:50592","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Accept-Encoding":["gzip"],"Cf-Ipcountry":["AU"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22"],"Cf-Request-Id":["09f81c940700001aaca3904000000001"],"Cf-Ray":["64d396ccd9631aac-SIN"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Connection":["Keep-Alive"],"X-Forwarded-For":["106.69.185.51"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Referer":["https://test.udance.com.au/"],"Cf-Connecting-Ip":["106.69.185.51"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006602499,"status":502,"err_id":"nfeudwy0e","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-10T21:42:11.888+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.165.126:18316","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/favicon.ico","headers":{"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"Referer":["https://test.udance.com.au/"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["106.69.185.51"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Pragma":["no-cache"],"Sec-Gpc":["1"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; cf_ob_info=502:64d396ccd9631aac:SIN; cf_use_ob=0"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Cf-Connecting-Ip":["106.69.185.51"],"Cf-Ipcountry":["AU"],"Cf-Ray":["64d396e34aea1aac-SIN"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["09f81ca21200001aacac959000000001"],"Connection":["Keep-Alive"],"Cache-Control":["no-cache"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005877493,"status":502,"err_id":"0vkbgp8eh","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}

Backend Caddy log extract

{"level":"info","ts":"2021-05-10T21:41:50.878+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:50.879+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:50.887+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003347e0"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654111.0122516,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654111.0166075,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002f2700"}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-10T21:41:51.023+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-10T21:41:51.045+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=91239) - Caddy is running in the background

Wait a minute… Do you have the ca-certificates package installed? Or maybe that’s stored in a different location on your system… You’re on a BSD right?

Edit: Well I’m going to guess that since you’re on FreeBSD, the smallstep trust install command doesn’t know what to do which is why we see “trust not supported”. I’ll need to dig into that to find out when they message is written.

From a bit of Googling, looks like FreeBSD now uses Mozilla’s trust store (aka NSS) as the system trust. This seems to be shipped as a single file that apps are meant to read from. Idk what to suggest, cause I’m not a BSD user.

Aha - yeah, the trust tool doesn’t seem to support FreeBSD.

If you could comment on the issue and help move it along, that’d be cool. If you can figure out the right procedure to install a custom root cert on FreeBSD and help them test it, changes there should eventually land in Caddy. :+1:

I will make a note but the absolute path depends on a few things. For example, in my Wiki setup I run caddy with caddy start in /root which makes the absolute path /root/.local/share/caddy/pki/authorities/local but when you use systemd the path would be (for at least Debian) /var/lib/caddy/.local/share/caddy/pki/authorities/local

:sob:

Okay (after regaining some composure), I must confess to being bamboozled at this point as this is now starting to go over my head. I’m having some difficulty connecting the dots.I don’t pretend to understand any of this stuff and what follows is just a little detective work on my part to see if anything has slipped through the cracks.

The last entry for the above issue was dated 10 April 2020. Seven days later on 17 April 2020 step-certificates and step-cli were ported to FreeBSD. Does this imply anything about the trust tool and FreeBSD, or am I barking up the wrong tree? I’m hoping this ignites a spark of hope. Assuming it does, are step-certificates and step-cli package dependencies for Caddy, and, with them installed, it may be possible to skirt around the ‘trust not supported’ error?

I’ve made a few inroads. It seems that Caddy can use its own internal CA or use a local step-ca instance. This is described in the Smallstep topic Configure popular ACME clients to use a private CA with the ACME protocol - Caddy V2.

When I compare the notes in that link with @Rob789’s wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts , it appears to match his backend configuration.

While I’ve managed to successfully set up the Smallstep CA in the same space as my Caddy frontend, what’s not clear is how I configure the Caddy frontend to work with the Smallstep CA rather than its internal CA.

I suspect this Caddy block needs to be modified to stitch the two together, but I’m not quite sure how to move forward from here? Help!

# ACME Server
caddy.roadrunner {
        acme_server
        tls internal
}

How I understood the Caddy’s local CA implementation is that it is using the Smallstep library. Thus, defining tls internal automatically makes use of the Smallstep CA. I don’t think Caddy has it’s own implementation for a certificate authority.

You can find more info on Github here.

Hi @Rob789,

I should have been clearer … Caddy can use its own internal CA (based on the Smallstep library) or use a local step-ca instance

Here’s an extract from the Smallstep reference I linked to…

Caddy comes with its own ACME server and by default it will generate an internal CA and issue certificates to itself. But, you can configure Caddy to use a local step-ca instance to obtain certificates.

Thanks for the Gihub link though. Makes for interesting reading.

If that’s possible, I’m not aware of it. But it would only be via JSON if it is. The facilities for the pki app are pretty limited from the Caddyfile.

/cc @matt

@francislavoie I’ve come full circle back to issue you identified in post #38 Bypassing the Caddy internal CA and using the Smallstep CA instead produces the same error. This Smallstep discussion has further detail.

So, for the moment, my mTLS journey has stalled :disappointed:. Until Smallstep fully support FreeBSD, mTLS via Caddy isn’t available for the FreeBSD platform.

Like I said earlier, you don’t have to use caddy trust for it to work, you just need to install the root CA certificate whatever way the operating system expects you to do it. You have direct access to the CA cert, so you can copy it wherever it needs to be. The caddy trust command is just convenience to automate that process.

I don’t know enough about FreeBSD to know the right approach to actually install the cert into the system trust, but again, Google is your friend. I’m sure you can figure it out :sweat_smile:

And once you have figured out the process, you can comment on the issue on the smallstep repo to help them add support for FreeBSD. Then everyone wins.

2 Likes

Well, that was the kick in the behind I needed to keep going for a bit longer, though I do have to say I’ve been out of my comfort zone for a while now. After trying a whole bunch of stuff that led nowhere, I might have made a little progress now. It turns out there are two issues I was grappling with:

  1. The trust store issue
  2. The ‘real’ root.crt location.

It turns out the second issue had a bearing on the first.

root.crt location

I was beating my head against the wall installing the root cert under /var/lib/caddy… into the trust store and not making any headway with mTLS.

The clue here was this note from @Rob789’s wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts :

NOTE: The absolute path for the certificate depends on how you run Caddy. Typically when you use systemd the absolute path will be /var/lib/caddy/.local/share/caddy/pki/authorities/local but if you run caddy with caddy start the path .local/share/caddy/pki/authorities/local

@Rob789 I’m curious to know where you came across this gem? Had I heeded the advice, my head might not have been as battered and bloodied :wink:

This is what made me change tack. When I looked more closely at both locations this is what I saw:

root@caddy:~ # ls -l /.local/share/caddy/pki/authorities/local/
total 26
-rw-------  1 root  wheel  676 May 19 12:27 intermediate.crt
-rw-------  1 root  wheel  227 May 19 12:27 intermediate.key
-rw-------  1 root  wheel  627 May  2 13:35 root.crt
-rw-------  1 root  wheel  227 May  2 13:35 root.key

root@caddy:~ # ls -l /var/lib/caddy/.local/share/caddy/pki/authorities/local/
total 26
-rw-------  1 root  wheel  676 May 10 11:06 intermediate.crt
-rw-------  1 root  wheel  227 May 10 11:06 intermediate.key
-rw-------  1 root  wheel  627 May 10 11:06 root.crt
-rw-------  1 root  wheel  227 May 10 11:06 root.key

There was a more recent date for the intermediate cert in the former, which led me to suspect that I needed to install the root cert from that directory in the trust store. As caddy start is used to run caddy in the environment, based on the note, this reinforced which root.crt I needed to install in the trust store. Once I understood this, I began to make some progress. mTLS is still not working completely though, but I’ll get to that shortly.

Trust root CA cert

So, I found a couple of resources that seemed to be pertinent.

  1. How to install a private CA certificate on FreeBSD, and more recently…
  2. Howto install a private CA certificate? - a FreeBSD forum thread.

Combining elements of both, I think I have installed the correct cert in the system trust. Here’s a log of the steps I took:

root@caddy:~ # freebsd-version
12.2-RELEASE-p6
root@caddy:~ # cp /.local/share/caddy/pki/authorities/local/root.crt /etc/ssl/certs
root@caddy:~ # openssl rehash /etc/ssl/certs
root@caddy:~ # openssl s_client -connect acme.lan:443 | grep -i -e verify
depth=1 CN = Caddy Local Authority - ECC Intermediate
verify return:1
depth=0
verify return:1
Verify return code: 0 (ok)

    Verify return code: 0 (ok)
root@caddy:~ #

Restarting the frontend Caddy service and checking the console log, there’s still the trust store error, but I assume that’s to be expected.

root@caddy:~ # tail --lines=20 /var/log/caddy.log
{"level":"info","ts":"2021-05-19T16:15:55.047+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00033a7e0"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412155.1001155,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412155.1057749,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000294690"}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-19T16:16:01.484+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/19 16:16:01 Note: NSS support is not available on your platform
2021/05/19 16:16:01 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-19T16:16:01.485+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","readymcgetty.com.au","xenografix.com.au","www.caffigoalkeeping.com","acme.lan","www.udance.com.au","udance.com.au","www.caffigoalkeeping.com.au","caffigoalkeeping.com.au","*.udance.com.au","caffigoalkeeping.com","www.readymcgetty.com.au"]}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-19T16:16:01.495+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=53177) - Caddy is running in the background
{"level":"info","ts":"2021-05-19T16:16:01.496+0800","logger":"tls","msg":"finished cleaning storage units"}

Restarting the backend Caddy service and checking the log, mTLS appears to be stirring into life!

{"level":"info","ts":"2021-05-19T16:22:28.408+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-19T16:22:28.408+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-19T16:22:28.411+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002e3110"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412548.539171,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412548.543643,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ec690"}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-19T16:22:28.550+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-19T16:22:28.572+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=28161) - Caddy is running in the background

Attempting to access test.udance.com.au, I see a 502 error in the browser, nothing in the backend Caddy log, but this error in the frontend log.

{"level":"error","ts":"2021-05-19T16:44:22.869+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"10.1.1.222:49381","proto":"HTTP/2.0","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wfwaf-authcookie-ce4a5970b2fd99774e0154e5e6b9716d=1%7Cadministrator%7Cd4b9b9e4e28bbee1e82f1b7f2b053da5fb2e984b668ed5b81d82bebf717d7436"],"Sec-Gpc":["1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005157315,"status":502,"err_id":"x10a03jus","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}

Have I installed the correct root cert in the system trust? I’m still not entirely sure. Can we get past this next stumbling block? That remains to be seen.

Relevant sections of the frontend Caddyfile…

...
# ACME server
acme.lan {
  acme_server
  tls internal
}
...
*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
  ...
    test         test.lan:443    yes    yes   yes        # test.udance.com.au
  ...
  route {
    ...
# Secure backend communication
    @mtls expression `{mtls} == "yes"`
    reverse_proxy @mtls {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
    ...
  }

Relevant sections of the backend Caddyfile (full copy in post #29) …

{
  ...
  acme_ca https://acme.lan/acme/local/directory
  acme_ca_root /etc/ssl/certs/root.crt
}

test.lan {
  ...
  }

Oh, once I worked out which root.crt to use, this appeared earlier in the day in the backend Caddy log. Seems promising.

{"level":"info","ts":"2021-05-19T14:39:19.704+0800","logger":"tls.renew","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:19.707+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"test.lan","remaining":-736110.707918771}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:20.069+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-19T14:39:20.138+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:44929","distributed":false}
{"level":"info","ts":"2021-05-19T14:39:20.492+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/BpA9sI3WDWa6fMtGGZufr5pRjKRKwo2l"}
{"level":"info","ts":"2021-05-19T14:39:20.704+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/gxvym5UeCVGotOR1rKJymjEj6AmDCq5X"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"releasing lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.706+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["test.lan"]}
{"level":"warn","ts":"2021-05-19T14:39:20.707+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T14:39:20.707+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["test.lan"],"new_expiration":"2021-05-19T18:39:20.000Z"}

It’s essentially the HOME environment variable of the user under which you ran Caddy.

Yeah, that’s harmless. Ignore it. In a later version, we’ll have an option to skip the install step which it tries at startup.

Did you also copy that newly found root cert to your backend Caddy instance too? It also needs that one to trust the frontend.

You might need to wipe the backend instance’s data storage to force it to fetch a new cert from the frontend with the right trust established maybe.

Yeah that looks pretty good. :thinking:

Maybe there’s another step you need to take to install the root cert, cause Caddy doesn’t use openssl - it uses the implementation of crypto from the Go standard library.

I feel like I read in some cursory Googling that FreeBSD uses Mozilla’s NSS tools as the trust store. Maybe there’s something else to be done there on the frontend.

Or, you could configure reverse_proxy’s http transport with tls_trusted_ca_certs instead of mucking with the trust store. Less nice if a solution though.

Yes, I did. :+1:

Can you elaborate, please? I’m not exactly sure what’s involved.

I’ve mucked around so much with Caddy and Smallstep in the same space, I might have messed something up. I think I’m going to start with a fresh Caddy instance and see if I end up in the same place. This will give me the opportunity to upgrade to Caddy 2.4.0. With a bit of luck, it may just be something I’ve inadvertently broken. If the issue is reproducible, I’ll then reflect on this advice.

Basically just rm -rf the contents of Caddy’s storage, then restart Caddy. But doing a fresh install avoids needing to do that :+1:

I rebuilt the frontend Caddy instance, but still end up with the ‘x509: certificate signed by unknown authority’ error…

root@caddy:~ # caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

root@caddy:~ # tail --lines=1 /var/log/caddy.log
{"level":"error","ts":"2021-05-20T04:46:11.333+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.21:60838","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["65202c54dc6dc81e-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["0a27fa09070000c81e109a5000000001"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006129711,"status":502,"err_id":"ghqnryw6i","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}

I deleted the backend Caddy storage, restarted Caddy and the log showed that a new cert was fetched from the frontend. Would this behaviour suggest that the root cert was correctly inserted in the system trust in the frontend Caddy instance?

{"level":"info","ts":"2021-05-20T04:24:26.146+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-20T04:24:26.147+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-20T04:24:26.149+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002bafc0"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621455866.2774534,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621455866.2815413,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002e48c0"}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"serving initial configuration"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls.obtain","msg":"acquiring lock","identifier":"test.lan"}
Successfully started Caddy (pid=39197) - Caddy is running in the background
{"level":"info","ts":"2021-05-20T04:24:26.309+0800","logger":"tls.obtain","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:27.120+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"tls-alpn-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-20T04:24:27.167+0800","logger":"tls","msg":"served key authentication certificate","server_name":"test.lan","challenge":"tls-alpn-01","remote":"10.1.1.4:26438","distributed":false}
{"level":"info","ts":"2021-05-20T04:24:27.666+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/m9Oisrd24B6ktWgqfg1lxbLamzzuEFY3"}
{"level":"info","ts":"2021-05-20T04:24:27.957+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/i9QC6UPnO0BdLB2U1QP1LSmhL0WO5OA1"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"releasing lock","identifier":"test.lan"}
{"level":"warn","ts":"2021-05-20T04:24:27.960+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}

If backend behaviour suggests that the cert was correctly installed in the frontend system trust, is it time to comment in the Smallstep repo?

ca_root_nss is one of the installed packages.

root@caddy:~ # pkg info
bash-5.1.4_1                   GNU Project's Bourne Again SHell
ca_root_nss-3.63               Root certificate bundle from the Mozilla Project
curl-7.76.0                    Command line tool and library for transferring data with URLs
cvsps-2.1_2                    Create patchset information from CVS
expat-2.2.10                   XML 1.0 parser written in C
gettext-runtime-0.21           GNU gettext runtime libraries and programs
git-2.31.1_1                   Distributed source code management tool
go-1.16.4,1                    Go programming language
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libnghttp2-1.43.0              HTTP/2.0 C Library
nano-5.5                       Nano's ANOther editor, an enhanced free Pico clone
p5-Authen-SASL-2.16_1          Perl5 module for SASL authentication
p5-CGI-4.51                    Handle Common Gateway Interface requests and responses
p5-Clone-0.45                  Recursively copy Perl datatypes
p5-Digest-HMAC-1.03_1          Perl5 interface to HMAC Message-Digest Algorithms
p5-Encode-Locale-1.05          Determine the locale encoding
p5-Error-0.17029               Error/exception handling in object-oriented programming style
p5-GSSAPI-0.28_1               Perl extension providing access to the GSSAPIv2 library
p5-HTML-Parser-3.75            Perl5 module for parsing HTML documents
p5-HTML-Tagset-3.20_1          Some useful data table in parsing HTML
p5-HTTP-Date-6.05              Conversion routines for the HTTP protocol date formats
p5-HTTP-Message-6.28           Representation of HTTP style messages
p5-IO-HTML-1.001_1             Open an HTML file with automatic charset detection
p5-IO-Socket-INET6-2.72_1      Perl module with object interface to AF_INET6 domain sockets
p5-IO-Socket-SSL-2.070         Perl5 interface to SSL sockets
p5-LWP-MediaTypes-6.04         Guess media type for a file or a URL
p5-Mozilla-CA-20200520         Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-SSLeay-1.88             Perl5 interface to SSL
p5-Socket6-0.29                IPv6 related part of the C socket.h defines and structure manipulators
p5-TimeDate-2.33,1             Perl5 module containing a better/faster date parser for absolute dates
p5-URI-5.07                    Perl5 interface to Uniform Resource Identifier (URI) references
pcre2-10.36                    Perl Compatible Regular Expressions library, version 2
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.16.3                     Package manager
python37-3.7.10                Interpreted object-oriented programming language
readline-8.1.0                 Library for editing command lines as they are typed

Is it time to consider this, or exhaust other options first e.g. try to understand and resolve the observed frontend error?

When comparing the frontend and backend Caddy logs, it’s pretty clear that the frontend is not communicating with the backend when the error occurs.

No, that just means that you used the right certificate for acme_ca_root on the backend. The frontend’s system trust is not involved when actually issuing certificates to other Caddy instances. It’s involved when actually proxying requests to the other instances.

It’s time to comment there as soon as you have something useful to report to them in terms of how best to install root CA certs on FreeBSD.

Good to know. So you’ll need to figure out how to add your own certs to that bundle.

I mean, you can try it at any point. If it works, then you know that it’s an option.

Right, cause the TLS handshake fails before the backend decides to log anything about the request. If you enable the debug global option, you might see some logs on the backend though.

Well, 16 days later and we have liftoff! :cowboy_hat_face:…sort of :thinking:

The clues that led me to a ‘solution’…
I looked a little deeper into ca_root_nss:

root@caddy:~ # pkg info -l ca_root_nss
ca_root_nss-3.63:
        /etc/ssl/cert.pem
        /usr/local/etc/ssl/cert.pem.sample
        /usr/local/openssl/cert.pem.sample
        /usr/local/share/certs/ca-root-nss.crt
        /usr/local/share/licenses/ca_root_nss-3.63/LICENSE
        /usr/local/share/licenses/ca_root_nss-3.63/MPL20
        /usr/local/share/licenses/ca_root_nss-3.63/catalog.mk

…and then appended root.crt to ca-root-nss.crt:

root@caddy:~ # cat /.local/share/caddy/pki/authorities/local/root.crt >> /usr/local/share/certs/ca-root-nss.crt

I then restarted frontend Caddy and noticed a change in the log regarding the trust of the root cert:

{"level":"info","ts":"2021-05-20T13:13:14.114+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-20T13:13:14.114+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-20T13:13:14.117+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000315d50"}
{"level":"info","ts":"2021-05-20T13:13:14.118+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T13:13:14.118+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621487594.170676,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621487594.1781077,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-20T13:13:14.181+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-20T13:13:14.181+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ce5b0"}
{"level":"info","ts":"2021-05-20T13:13:14.191+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:13:14.191+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["xenografix.com.au","acme.lan","www.readymcgetty.com.au","www.caffigoalkeeping.com.au","readymcgetty.com.au","www.udance.com.au","udance.com.au","www.xenografix.com.au","caffigoalkeeping.com","*.udance.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com.au"]}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-20T13:13:20.994+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:13:20.997+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-20T13:13:21.001+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:13:21.001+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=64612) - Caddy is running in the background

Accessing the test site… Nice! :smiley:


However, I don’t see anything in the frontend or backend Caddy logs unless I turn debug on. It would be nice to have some basic logging happening for mTLS.

The system trust solution I stumbled upon is semi-permanent at best. The clue was in this FreeBSD bug report Bug 160387 - security/ca_root_nss: Allow user to trust extra local certificates. The trust breaks if the ca_root_nss package is updated. To fix it, it’s necessary to append root.crt to ca-root-nss.crt again. For reasons described in the bug report, a permanent solution is unlikely to be forthcoming.

Question: How do I convince myself that the communication path between the backend and frontend Caddy services is actually encrypted?

:rocket::rocket::rocket::rocket::rocket::rocket::rocket::rocket:

The reverse_proxy logs should show that the request has TLS on it. If it does, then you’re good to go. Not much else to say, really. There’s nothing inherent about mTLS that can be logged, it’s just regular HTTPS proxying, plus non-public trust. That’s all.

Figures. That’s kinda annoying. There might be a way to hook into your package manager to run some script after each time a package is updated… maybe? I know nothing of FreeBSD’s package manager.

But you could report this on the smallstep repo. Should be helpful.

The euphoria was somewhat short-lived. I’m no longer able to access the test site. :sob:

In the few hours I’ve been away, this is what’s accumulated in the logs:

Backend Caddy log:

{"level":"info","ts":"2021-05-20T13:39:29.599+0800","logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:52184","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1684"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":"2021-05-20T13:39:29.600+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-20T13:39:29.601+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:39:29.601+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:39:29.603+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"warn","ts":"2021-05-20T13:39:29.604+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:39:29.604+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ca700"}
{"level":"info","ts":"2021-05-20T13:39:30.662+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002c5570"}
{"level":"info","ts":"2021-05-20T13:39:30.663+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:39:30.663+0800","logger":"admin.api","msg":"load complete"}
{"level":"info","ts":"2021-05-20T13:39:31.208+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T20:39:29.606+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["test.lan"],"remaining":14017.393058953}
{"level":"info","ts":"2021-05-20T20:39:29.607+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["test.lan"],"remaining":14017.392423428}
{"level":"info","ts":"2021-05-20T20:39:29.608+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:29.640+0800","logger":"tls.renew","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:29.640+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"test.lan","remaining":14017.359107754}
{"level":"info","ts":"2021-05-20T20:39:29.642+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T20:39:29.642+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T20:39:29.998+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-20T20:39:30.055+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:47482","distributed":false}
{"level":"info","ts":"2021-05-20T20:39:30.404+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri"}
{"level":"info","ts":"2021-05-20T20:39:30.583+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/fPbBOhtqkDfWJOdMMvB2B6kKNVTeAvAQ"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls.renew","msg":"releasing lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["test.lan"]}
{"level":"warn","ts":"2021-05-20T20:39:30.586+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T20:39:30.586+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["test.lan"],"new_expiration":"2021-05-21T00:39:30.000Z"}

Frontend Caddy log - noticeable events.

{"level":"info","ts":"2021-05-20T13:39:55.408+0800","logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:27757","headers":{"Accept-Encoding":["gzip"],"Content-Length":["10520"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":"2021-05-20T13:39:55.412+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-20T13:39:55.412+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003060e0"}
{"level":"info","ts":"2021-05-20T13:39:55.413+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:39:55.413+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:40:01.692+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.caffigoalkeeping.com.au","udance.com.au","www.readymcgetty.com.au","www.xenografix.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","readymcgetty.com.au","acme.lan","www.caffigoalkeeping.com","caffigoalkeeping.com","xenografix.com.au"]}
{"level":"warn","ts":"2021-05-20T13:40:01.699+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:40:01.699+0800","logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000178460"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","logger":"admin.api","msg":"load complete"}
{"level":"info","ts":"2021-05-20T13:40:02.344+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}

...

{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["acme.lan"],"remaining":14226.584475591}
{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["acme.lan"],"remaining":14226.584391609}
{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.471+0800","logger":"tls.renew","msg":"lock acquired","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.471+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"acme.lan","remaining":14226.528197201}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls.renew","msg":"releasing lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["acme.lan"]}
{"level":"warn","ts":"2021-05-20T19:49:55.474+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T19:49:55.474+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["acme.lan"],"new_expiration":"2021-05-20T23:49:55.000Z"}

...

2021/05/20 20:39:29 {"newNonce":"https://acme.lan/acme/local/new-nonce","newAccount":"https://acme.lan/acme/local/new-account","newOrder":"https://acme.lan/acme/local/new-order","revokeCert":"https://acme.lan/acme/local/revoke-cert","keyChange":"https://acme.lan/acme/local/key-change"}
2021/05/20 20:39:29 {"status":"pending","expires":"2021-05-21T12:39:30Z","identifiers":[{"type":"dns","value":"test.lan"}],"notBefore":"2021-05-20T12:39:30Z","notAfter":"2021-05-21T00:39:30Z","authorizations":["https://acme.lan/acme/local/authz/niNSkBvVy4bwYMeKQvS48pfBWzgw4DRY"],"finalize":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri/finalize"}
2021/05/20 20:39:29 {"identifier":{"type":"dns","value":"test.lan"},"status":"pending","expires":"2021-05-21T12:39:30Z","challenges":[{"type":"http-01","status":"pending","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"},{"type":"tls-alpn-01","status":"pending","token":"7tjZva3KtSzzdnGtK6YPOzNoasbiVcZW","url":"https://acme.lan/acme/local/challenge/yVYPnacjhH6QmzUhZpjNPOfTGPEQWHwV"},{"type":"dns-01","status":"pending","token":"CQzbgaNLQdMzir80pqYnQtX74U0tRNod","url":"https://acme.lan/acme/local/challenge/mBUvK1duKwkNj1VeQ3ysh4POvgKfqceU"}],"wildcard":false}
2021/05/20 20:39:30 {"type":"http-01","status":"valid","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","validated":"2021-05-20T12:39:30Z","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"}
2021/05/20 20:39:30 {"identifier":{"type":"dns","value":"test.lan"},"status":"valid","expires":"2021-05-21T12:39:30Z","challenges":[{"type":"http-01","status":"valid","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","validated":"2021-05-20T12:39:30Z","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"},{"type":"tls-alpn-01","status":"pending","token":"7tjZva3KtSzzdnGtK6YPOzNoasbiVcZW","url":"https://acme.lan/acme/local/challenge/yVYPnacjhH6QmzUhZpjNPOfTGPEQWHwV"},{"type":"dns-01","status":"pending","token":"CQzbgaNLQdMzir80pqYnQtX74U0tRNod","url":"https://acme.lan/acme/local/challenge/mBUvK1duKwkNj1VeQ3ysh4POvgKfqceU"}],"wildcard":false}
2021/05/20 20:39:30 {"status":"valid","expires":"2021-05-21T12:39:30Z","identifiers":[{"type":"dns","value":"test.lan"}],"notBefore":"2021-05-20T12:39:30Z","notAfter":"2021-05-21T00:39:30Z","authorizations":["https://acme.lan/acme/local/authz/niNSkBvVy4bwYMeKQvS48pfBWzgw4DRY"],"finalize":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri/finalize","certificate":"https://acme.lan/acme/local/certificate/fPbBOhtqkDfWJOdMMvB2B6kKNVTeAvAQ"}

...

There’s a pattern. In the frontend log, you can match up the first and third groups with what’s in the backend log. Interestingly, the format of the third group changes in the log, but then returns to normal in the messages that follow this group.I’m not sure if there’s any significance in this.

When I attempt to acces the site externally, I can a redirection from test.udance.com.au to test.lan.

What’s probably more telling is when I access test.udance.com.au from the local network (split DNS arrangement). There’s a privacy error and a message about an invalid CA.