mTLS under FreeBSD

My bad. That’s me breaking up long lines to copy across and, in this instance, forgetting to restore the line.

Good to know.

The first three related messages post-restart from the frontend log. There are a few more messages similar to this that follow, and then what appears to be a cert refresh, and then more error messages much like the first three.

...
{"level":"error","ts":"2021-05-09T23:00:37.235+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.21:22832","proto":"HTTP/2.0","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"Cf-Ray":["64cbcc60ddf29b97-DFW"],"X-Forwarded-Proto":["https"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cf-Request-Id":["09f33e108b00009b979e2cf000000001"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["192.0.91.177"],"Cf-Visitor":["{\"scheme\":\"https\"}"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006550785,"status":502,"err_id":"kx7sdf71h","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...
{"level":"error","ts":"2021-05-09T23:06:06.006+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"10.1.1.54:21726","proto":"HTTP/1.1","method":"POST","host":"test.udance.com.au","uri":"/wp-admin/admin-ajax.php?action=updraft_central","headers":{"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-admin/admin-ajax.php?action=updraft_central"],"Connection":["close"],"Content-Length":["1146"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.1; class-udrpc.php/1.4.21; https://blog.udance.com.au"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006877555,"status":502,"err_id":"jkzpr9x4p","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-09T23:07:06.187+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.118:47290","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Connection":["Keep-Alive"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["09f343ffe800000f3e8815d000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["64cbd5dfde3a0f3e-DFW"],"Cf-Connecting-Ip":["192.0.91.177"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006013134,"status":502,"err_id":"xv3yykkdr","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...
{"level":"info","ts":"2021-05-10T00:15:22.128+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["acme.lan"],"remaining":14096.871491679}
{"level":"info","ts":"2021-05-10T00:15:22.128+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["acme.lan"],"remaining":14096.871356348}
{"level":"info","ts":"2021-05-10T00:15:22.129+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.134+0800","logger":"tls.renew","msg":"lock acquired","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.135+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"acme.lan","remaining":14096.864875394}
{"level":"info","ts":"2021-05-10T00:15:22.137+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.137+0800","logger":"tls.renew","msg":"releasing lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.138+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["acme.lan"]}
{"level":"warn","ts":"2021-05-10T00:15:22.139+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T00:15:22.139+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["acme.lan"],"new_expiration":"2021-05-10T04:15:22.000Z"}
{"level":"error","ts":"2021-05-10T00:18:32.293+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.6:43830","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"Connection":["Keep-Alive"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["64cc3e840a3a2f25-DFW"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cf-Request-Id":["09f385668500002f25a01dc000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006488916,"status":502,"err_id":"pvzr5v6v0","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...

Where there’s an error in the log, I notice this at the end of the line

"err_trace":"reverseproxy.statusError (reverseproxy.go:850)"

Strange, nothing new added in the backend log.

1 Like

I think this is the relevant bit. So like I said earlier, I think you need to run caddy trust on the front-end. Basically the backend did get certificates signed by the front-end, but then when the front-end goes to proxy, it doesn’t trust the backend’s cert because the root CA cert isn’t in the front-end’s system trust store (which is used when making HTTPS requests with the proxy).

Alternatively you can configure transport http with the tls_trusted_ca_certs with the path to the root CA cert file (i.e. /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt), but that’s a bit of a dumb workaround. Running caddy trust is the better option.

Basically, just run this on your frontend server:

sudo HOME=/var/lib/caddy caddy trust

Kinda dumb, but it should do the trick.

The caddy trust command is not very intelligent right now… the trouble is the user it runs as. If you just run it with caddy trust as your own user then it’ll try to use your user’s home. If you just run it with sudo caddy trust then it’ll use the root user’s home (i.e. /root). But what we actually want is the home of the caddy user (under which the systemd service runs Caddy) which is /var/lib/caddy. Hence hacking it by overriding HOME in that command :roll_eyes:

I’ll see if that can be smoothed out…

This made me realise I’d copied over the incorrect certificate to the backend Caddy server. I had copied across /.local/share/caddy/pki/authorities/local/root.crt instead of /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt.

@Rob789 The above paragraph in your wiki article under the heading Backends should be updated to use the absolute path to avoid any ambiguity:

root@caddy:~ # sudo HOME=/var/lib/caddy caddy trust
2021/05/10 13:40:41.376 WARN    ca.local        installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2021/05/10 21:40:41 Note: NSS support is not available on your platform
2021/05/10 21:40:41 define JAVA_HOME environment variable to use the Java trust
trust: trust not supported

After copying across the correct cert to the backend server, I then restarted both frontend and backend Caddy services and attempted to access the test site. Sadly still no go :cry:

Frontend caddy log extract

{"level":"info","ts":"2021-05-10T21:41:44.437+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:44.437+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:44.443+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00035a690"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654104.5705228,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654104.6052015,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:44.613+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-10T21:41:44.614+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00032ff80"}
{"level":"info","ts":"2021-05-10T21:41:44.644+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:44.645+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-10T21:41:57.225+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/10 21:41:57 Note: NSS support is not available on your platform
2021/05/10 21:41:57 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-10T21:41:57.226+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","readymcgetty.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","udance.com.au","www.caffigoalkeeping.com.au","www.readymcgetty.com.au","xenografix.com.au","acme.lan"]}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-10T21:41:57.258+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=96255) - Caddy is running in the background
{"level":"info","ts":"2021-05-10T21:41:57.260+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"error","ts":"2021-05-10T21:42:08.298+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.166.143:50592","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Accept-Encoding":["gzip"],"Cf-Ipcountry":["AU"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22"],"Cf-Request-Id":["09f81c940700001aaca3904000000001"],"Cf-Ray":["64d396ccd9631aac-SIN"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Connection":["Keep-Alive"],"X-Forwarded-For":["106.69.185.51"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Referer":["https://test.udance.com.au/"],"Cf-Connecting-Ip":["106.69.185.51"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006602499,"status":502,"err_id":"nfeudwy0e","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-10T21:42:11.888+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.165.126:18316","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/favicon.ico","headers":{"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"Referer":["https://test.udance.com.au/"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["106.69.185.51"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Pragma":["no-cache"],"Sec-Gpc":["1"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; cf_ob_info=502:64d396ccd9631aac:SIN; cf_use_ob=0"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Cf-Connecting-Ip":["106.69.185.51"],"Cf-Ipcountry":["AU"],"Cf-Ray":["64d396e34aea1aac-SIN"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["09f81ca21200001aacac959000000001"],"Connection":["Keep-Alive"],"Cache-Control":["no-cache"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005877493,"status":502,"err_id":"0vkbgp8eh","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}

Backend Caddy log extract

{"level":"info","ts":"2021-05-10T21:41:50.878+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:50.879+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:50.887+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003347e0"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654111.0122516,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654111.0166075,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002f2700"}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-10T21:41:51.023+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-10T21:41:51.045+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=91239) - Caddy is running in the background

Wait a minute… Do you have the ca-certificates package installed? Or maybe that’s stored in a different location on your system… You’re on a BSD right?

Edit: Well I’m going to guess that since you’re on FreeBSD, the smallstep trust install command doesn’t know what to do which is why we see “trust not supported”. I’ll need to dig into that to find out when they message is written.

From a bit of Googling, looks like FreeBSD now uses Mozilla’s trust store (aka NSS) as the system trust. This seems to be shipped as a single file that apps are meant to read from. Idk what to suggest, cause I’m not a BSD user.

Aha - yeah, the trust tool doesn’t seem to support FreeBSD.

If you could comment on the issue and help move it along, that’d be cool. If you can figure out the right procedure to install a custom root cert on FreeBSD and help them test it, changes there should eventually land in Caddy. :+1:

I will make a note but the absolute path depends on a few things. For example, in my Wiki setup I run caddy with caddy start in /root which makes the absolute path /root/.local/share/caddy/pki/authorities/local but when you use systemd the path would be (for at least Debian) /var/lib/caddy/.local/share/caddy/pki/authorities/local

:sob:

Okay (after regaining some composure), I must confess to being bamboozled at this point as this is now starting to go over my head. I’m having some difficulty connecting the dots.I don’t pretend to understand any of this stuff and what follows is just a little detective work on my part to see if anything has slipped through the cracks.

The last entry for the above issue was dated 10 April 2020. Seven days later on 17 April 2020 step-certificates and step-cli were ported to FreeBSD. Does this imply anything about the trust tool and FreeBSD, or am I barking up the wrong tree? I’m hoping this ignites a spark of hope. Assuming it does, are step-certificates and step-cli package dependencies for Caddy, and, with them installed, it may be possible to skirt around the ‘trust not supported’ error?

I’ve made a few inroads. It seems that Caddy can use its own internal CA or use a local step-ca instance. This is described in the Smallstep topic Configure popular ACME clients to use a private CA with the ACME protocol - Caddy V2.

When I compare the notes in that link with @Rob789’s wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts , it appears to match his backend configuration.

While I’ve managed to successfully set up the Smallstep CA in the same space as my Caddy frontend, what’s not clear is how I configure the Caddy frontend to work with the Smallstep CA rather than its internal CA.

I suspect this Caddy block needs to be modified to stitch the two together, but I’m not quite sure how to move forward from here? Help!

# ACME Server
caddy.roadrunner {
        acme_server
        tls internal
}

How I understood the Caddy’s local CA implementation is that it is using the Smallstep library. Thus, defining tls internal automatically makes use of the Smallstep CA. I don’t think Caddy has it’s own implementation for a certificate authority.

You can find more info on Github here.

Hi @Rob789,

I should have been clearer … Caddy can use its own internal CA (based on the Smallstep library) or use a local step-ca instance

Here’s an extract from the Smallstep reference I linked to…

Caddy comes with its own ACME server and by default it will generate an internal CA and issue certificates to itself. But, you can configure Caddy to use a local step-ca instance to obtain certificates.

Thanks for the Gihub link though. Makes for interesting reading.

If that’s possible, I’m not aware of it. But it would only be via JSON if it is. The facilities for the pki app are pretty limited from the Caddyfile.

/cc @matt

@francislavoie I’ve come full circle back to issue you identified in post #38 Bypassing the Caddy internal CA and using the Smallstep CA instead produces the same error. This Smallstep discussion has further detail.

So, for the moment, my mTLS journey has stalled :disappointed:. Until Smallstep fully support FreeBSD, mTLS via Caddy isn’t available for the FreeBSD platform.

Like I said earlier, you don’t have to use caddy trust for it to work, you just need to install the root CA certificate whatever way the operating system expects you to do it. You have direct access to the CA cert, so you can copy it wherever it needs to be. The caddy trust command is just convenience to automate that process.

I don’t know enough about FreeBSD to know the right approach to actually install the cert into the system trust, but again, Google is your friend. I’m sure you can figure it out :sweat_smile:

And once you have figured out the process, you can comment on the issue on the smallstep repo to help them add support for FreeBSD. Then everyone wins.

2 Likes

Well, that was the kick in the behind I needed to keep going for a bit longer, though I do have to say I’ve been out of my comfort zone for a while now. After trying a whole bunch of stuff that led nowhere, I might have made a little progress now. It turns out there are two issues I was grappling with:

  1. The trust store issue
  2. The ‘real’ root.crt location.

It turns out the second issue had a bearing on the first.

root.crt location

I was beating my head against the wall installing the root cert under /var/lib/caddy… into the trust store and not making any headway with mTLS.

The clue here was this note from @Rob789’s wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts :

NOTE: The absolute path for the certificate depends on how you run Caddy. Typically when you use systemd the absolute path will be /var/lib/caddy/.local/share/caddy/pki/authorities/local but if you run caddy with caddy start the path .local/share/caddy/pki/authorities/local

@Rob789 I’m curious to know where you came across this gem? Had I heeded the advice, my head might not have been as battered and bloodied :wink:

This is what made me change tack. When I looked more closely at both locations this is what I saw:

root@caddy:~ # ls -l /.local/share/caddy/pki/authorities/local/
total 26
-rw-------  1 root  wheel  676 May 19 12:27 intermediate.crt
-rw-------  1 root  wheel  227 May 19 12:27 intermediate.key
-rw-------  1 root  wheel  627 May  2 13:35 root.crt
-rw-------  1 root  wheel  227 May  2 13:35 root.key

root@caddy:~ # ls -l /var/lib/caddy/.local/share/caddy/pki/authorities/local/
total 26
-rw-------  1 root  wheel  676 May 10 11:06 intermediate.crt
-rw-------  1 root  wheel  227 May 10 11:06 intermediate.key
-rw-------  1 root  wheel  627 May 10 11:06 root.crt
-rw-------  1 root  wheel  227 May 10 11:06 root.key

There was a more recent date for the intermediate cert in the former, which led me to suspect that I needed to install the root cert from that directory in the trust store. As caddy start is used to run caddy in the environment, based on the note, this reinforced which root.crt I needed to install in the trust store. Once I understood this, I began to make some progress. mTLS is still not working completely though, but I’ll get to that shortly.

Trust root CA cert

So, I found a couple of resources that seemed to be pertinent.

  1. How to install a private CA certificate on FreeBSD, and more recently…
  2. Howto install a private CA certificate? - a FreeBSD forum thread.

Combining elements of both, I think I have installed the correct cert in the system trust. Here’s a log of the steps I took:

root@caddy:~ # freebsd-version
12.2-RELEASE-p6
root@caddy:~ # cp /.local/share/caddy/pki/authorities/local/root.crt /etc/ssl/certs
root@caddy:~ # openssl rehash /etc/ssl/certs
root@caddy:~ # openssl s_client -connect acme.lan:443 | grep -i -e verify
depth=1 CN = Caddy Local Authority - ECC Intermediate
verify return:1
depth=0
verify return:1
Verify return code: 0 (ok)

    Verify return code: 0 (ok)
root@caddy:~ #

Restarting the frontend Caddy service and checking the console log, there’s still the trust store error, but I assume that’s to be expected.

root@caddy:~ # tail --lines=20 /var/log/caddy.log
{"level":"info","ts":"2021-05-19T16:15:55.047+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00033a7e0"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412155.1001155,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412155.1057749,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000294690"}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-19T16:16:01.484+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/19 16:16:01 Note: NSS support is not available on your platform
2021/05/19 16:16:01 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-19T16:16:01.485+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","readymcgetty.com.au","xenografix.com.au","www.caffigoalkeeping.com","acme.lan","www.udance.com.au","udance.com.au","www.caffigoalkeeping.com.au","caffigoalkeeping.com.au","*.udance.com.au","caffigoalkeeping.com","www.readymcgetty.com.au"]}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-19T16:16:01.495+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=53177) - Caddy is running in the background
{"level":"info","ts":"2021-05-19T16:16:01.496+0800","logger":"tls","msg":"finished cleaning storage units"}

Restarting the backend Caddy service and checking the log, mTLS appears to be stirring into life!

{"level":"info","ts":"2021-05-19T16:22:28.408+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-19T16:22:28.408+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-19T16:22:28.411+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002e3110"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412548.539171,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412548.543643,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ec690"}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-19T16:22:28.550+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-19T16:22:28.572+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=28161) - Caddy is running in the background

Attempting to access test.udance.com.au, I see a 502 error in the browser, nothing in the backend Caddy log, but this error in the frontend log.

{"level":"error","ts":"2021-05-19T16:44:22.869+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"10.1.1.222:49381","proto":"HTTP/2.0","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wfwaf-authcookie-ce4a5970b2fd99774e0154e5e6b9716d=1%7Cadministrator%7Cd4b9b9e4e28bbee1e82f1b7f2b053da5fb2e984b668ed5b81d82bebf717d7436"],"Sec-Gpc":["1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005157315,"status":502,"err_id":"x10a03jus","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}

Have I installed the correct root cert in the system trust? I’m still not entirely sure. Can we get past this next stumbling block? That remains to be seen.

Relevant sections of the frontend Caddyfile…

...
# ACME server
acme.lan {
  acme_server
  tls internal
}
...
*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
  ...
    test         test.lan:443    yes    yes   yes        # test.udance.com.au
  ...
  route {
    ...
# Secure backend communication
    @mtls expression `{mtls} == "yes"`
    reverse_proxy @mtls {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
    ...
  }

Relevant sections of the backend Caddyfile (full copy in post #29) …

{
  ...
  acme_ca https://acme.lan/acme/local/directory
  acme_ca_root /etc/ssl/certs/root.crt
}

test.lan {
  ...
  }

Oh, once I worked out which root.crt to use, this appeared earlier in the day in the backend Caddy log. Seems promising.

{"level":"info","ts":"2021-05-19T14:39:19.704+0800","logger":"tls.renew","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:19.707+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"test.lan","remaining":-736110.707918771}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:20.069+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-19T14:39:20.138+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:44929","distributed":false}
{"level":"info","ts":"2021-05-19T14:39:20.492+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/BpA9sI3WDWa6fMtGGZufr5pRjKRKwo2l"}
{"level":"info","ts":"2021-05-19T14:39:20.704+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/gxvym5UeCVGotOR1rKJymjEj6AmDCq5X"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"releasing lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.706+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["test.lan"]}
{"level":"warn","ts":"2021-05-19T14:39:20.707+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T14:39:20.707+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["test.lan"],"new_expiration":"2021-05-19T18:39:20.000Z"}

It’s essentially the HOME environment variable of the user under which you ran Caddy.

Yeah, that’s harmless. Ignore it. In a later version, we’ll have an option to skip the install step which it tries at startup.

Did you also copy that newly found root cert to your backend Caddy instance too? It also needs that one to trust the frontend.

You might need to wipe the backend instance’s data storage to force it to fetch a new cert from the frontend with the right trust established maybe.

Yeah that looks pretty good. :thinking:

Maybe there’s another step you need to take to install the root cert, cause Caddy doesn’t use openssl - it uses the implementation of crypto from the Go standard library.

I feel like I read in some cursory Googling that FreeBSD uses Mozilla’s NSS tools as the trust store. Maybe there’s something else to be done there on the frontend.

Or, you could configure reverse_proxy’s http transport with tls_trusted_ca_certs instead of mucking with the trust store. Less nice if a solution though.

Yes, I did. :+1:

Can you elaborate, please? I’m not exactly sure what’s involved.

I’ve mucked around so much with Caddy and Smallstep in the same space, I might have messed something up. I think I’m going to start with a fresh Caddy instance and see if I end up in the same place. This will give me the opportunity to upgrade to Caddy 2.4.0. With a bit of luck, it may just be something I’ve inadvertently broken. If the issue is reproducible, I’ll then reflect on this advice.

Basically just rm -rf the contents of Caddy’s storage, then restart Caddy. But doing a fresh install avoids needing to do that :+1:

I rebuilt the frontend Caddy instance, but still end up with the ‘x509: certificate signed by unknown authority’ error…

root@caddy:~ # caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

root@caddy:~ # tail --lines=1 /var/log/caddy.log
{"level":"error","ts":"2021-05-20T04:46:11.333+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.21:60838","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["65202c54dc6dc81e-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["0a27fa09070000c81e109a5000000001"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006129711,"status":502,"err_id":"ghqnryw6i","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}

I deleted the backend Caddy storage, restarted Caddy and the log showed that a new cert was fetched from the frontend. Would this behaviour suggest that the root cert was correctly inserted in the system trust in the frontend Caddy instance?

{"level":"info","ts":"2021-05-20T04:24:26.146+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-20T04:24:26.147+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-20T04:24:26.149+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002bafc0"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621455866.2774534,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621455866.2815413,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002e48c0"}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"serving initial configuration"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls.obtain","msg":"acquiring lock","identifier":"test.lan"}
Successfully started Caddy (pid=39197) - Caddy is running in the background
{"level":"info","ts":"2021-05-20T04:24:26.309+0800","logger":"tls.obtain","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:27.120+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"tls-alpn-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-20T04:24:27.167+0800","logger":"tls","msg":"served key authentication certificate","server_name":"test.lan","challenge":"tls-alpn-01","remote":"10.1.1.4:26438","distributed":false}
{"level":"info","ts":"2021-05-20T04:24:27.666+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/m9Oisrd24B6ktWgqfg1lxbLamzzuEFY3"}
{"level":"info","ts":"2021-05-20T04:24:27.957+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/i9QC6UPnO0BdLB2U1QP1LSmhL0WO5OA1"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"releasing lock","identifier":"test.lan"}
{"level":"warn","ts":"2021-05-20T04:24:27.960+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}

If backend behaviour suggests that the cert was correctly installed in the frontend system trust, is it time to comment in the Smallstep repo?

ca_root_nss is one of the installed packages.

root@caddy:~ # pkg info
bash-5.1.4_1                   GNU Project's Bourne Again SHell
ca_root_nss-3.63               Root certificate bundle from the Mozilla Project
curl-7.76.0                    Command line tool and library for transferring data with URLs
cvsps-2.1_2                    Create patchset information from CVS
expat-2.2.10                   XML 1.0 parser written in C
gettext-runtime-0.21           GNU gettext runtime libraries and programs
git-2.31.1_1                   Distributed source code management tool
go-1.16.4,1                    Go programming language
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libnghttp2-1.43.0              HTTP/2.0 C Library
nano-5.5                       Nano's ANOther editor, an enhanced free Pico clone
p5-Authen-SASL-2.16_1          Perl5 module for SASL authentication
p5-CGI-4.51                    Handle Common Gateway Interface requests and responses
p5-Clone-0.45                  Recursively copy Perl datatypes
p5-Digest-HMAC-1.03_1          Perl5 interface to HMAC Message-Digest Algorithms
p5-Encode-Locale-1.05          Determine the locale encoding
p5-Error-0.17029               Error/exception handling in object-oriented programming style
p5-GSSAPI-0.28_1               Perl extension providing access to the GSSAPIv2 library
p5-HTML-Parser-3.75            Perl5 module for parsing HTML documents
p5-HTML-Tagset-3.20_1          Some useful data table in parsing HTML
p5-HTTP-Date-6.05              Conversion routines for the HTTP protocol date formats
p5-HTTP-Message-6.28           Representation of HTTP style messages
p5-IO-HTML-1.001_1             Open an HTML file with automatic charset detection
p5-IO-Socket-INET6-2.72_1      Perl module with object interface to AF_INET6 domain sockets
p5-IO-Socket-SSL-2.070         Perl5 interface to SSL sockets
p5-LWP-MediaTypes-6.04         Guess media type for a file or a URL
p5-Mozilla-CA-20200520         Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-SSLeay-1.88             Perl5 interface to SSL
p5-Socket6-0.29                IPv6 related part of the C socket.h defines and structure manipulators
p5-TimeDate-2.33,1             Perl5 module containing a better/faster date parser for absolute dates
p5-URI-5.07                    Perl5 interface to Uniform Resource Identifier (URI) references
pcre2-10.36                    Perl Compatible Regular Expressions library, version 2
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.16.3                     Package manager
python37-3.7.10                Interpreted object-oriented programming language
readline-8.1.0                 Library for editing command lines as they are typed

Is it time to consider this, or exhaust other options first e.g. try to understand and resolve the observed frontend error?

When comparing the frontend and backend Caddy logs, it’s pretty clear that the frontend is not communicating with the backend when the error occurs.

No, that just means that you used the right certificate for acme_ca_root on the backend. The frontend’s system trust is not involved when actually issuing certificates to other Caddy instances. It’s involved when actually proxying requests to the other instances.

It’s time to comment there as soon as you have something useful to report to them in terms of how best to install root CA certs on FreeBSD.

Good to know. So you’ll need to figure out how to add your own certs to that bundle.

I mean, you can try it at any point. If it works, then you know that it’s an option.

Right, cause the TLS handshake fails before the backend decides to log anything about the request. If you enable the debug global option, you might see some logs on the backend though.