mTLS under FreeBSD

[basil (Basil Hendroff)]

@francislavoie

If you just run it with caddy trust as your own user then it’ll try to use your user’s home. If you just run it with sudo caddy trust then it’ll use the root user’s home (i.e. /root). But what we actually want is the home of the caddy user (under which the systemd service runs Caddy) which is /var/lib/caddy.

This made me realise I’d copied over the incorrect certificate to the backend Caddy server. I had copied across /.local/share/caddy/pki/authorities/local/root.crt instead of /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt.

@Rob789

To request a certificate, a root certificate from the server is required. (Read this blog from Mike Malone to understand why). Once you have started the Caddy service on the frontend for the first time and there are no errors, a file named root.crt is generated in .local/share/caddy/pki/authorities/local

@Rob789 The above paragraph in your wiki article under the heading Backends should be updated to use the absolute path to avoid any ambiguity:

@Rob789

To request a certificate, a root certificate from the server is required. (Read this blog from Mike Malone to understand why). Once you have started the Caddy service on the frontend for the first time and there are no errors, a file named root.crt is generated in /var/lib/caddy/.local/share/caddy/pki/authorities/local

@francislavoie

Basically, just run this on your frontend server:

sudo HOME=/var/lib/caddy caddy trust

root@caddy:~ # sudo HOME=/var/lib/caddy caddy trust
2021/05/10 13:40:41.376 WARN    ca.local        installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2021/05/10 21:40:41 Note: NSS support is not available on your platform
2021/05/10 21:40:41 define JAVA_HOME environment variable to use the Java trust
trust: trust not supported

After copying across the correct cert to the backend server, I then restarted both frontend and backend Caddy services and attempted to access the test site. Sadly still no go

Frontend caddy log extract

{"level":"info","ts":"2021-05-10T21:41:44.437+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:44.437+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:44.443+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00035a690"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654104.5705228,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654104.6052015,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:44.613+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-10T21:41:44.614+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00032ff80"}
{"level":"info","ts":"2021-05-10T21:41:44.644+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:44.645+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-10T21:41:57.225+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/10 21:41:57 Note: NSS support is not available on your platform
2021/05/10 21:41:57 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-10T21:41:57.226+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","readymcgetty.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","udance.com.au","www.caffigoalkeeping.com.au","www.readymcgetty.com.au","xenografix.com.au","acme.lan"]}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-10T21:41:57.258+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=96255) - Caddy is running in the background
{"level":"info","ts":"2021-05-10T21:41:57.260+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"error","ts":"2021-05-10T21:42:08.298+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.166.143:50592","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Accept-Encoding":["gzip"],"Cf-Ipcountry":["AU"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22"],"Cf-Request-Id":["09f81c940700001aaca3904000000001"],"Cf-Ray":["64d396ccd9631aac-SIN"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Connection":["Keep-Alive"],"X-Forwarded-For":["106.69.185.51"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Referer":["https://test.udance.com.au/"],"Cf-Connecting-Ip":["106.69.185.51"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006602499,"status":502,"err_id":"nfeudwy0e","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-10T21:42:11.888+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.165.126:18316","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/favicon.ico","headers":{"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"Referer":["https://test.udance.com.au/"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["106.69.185.51"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Pragma":["no-cache"],"Sec-Gpc":["1"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; cf_ob_info=502:64d396ccd9631aac:SIN; cf_use_ob=0"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Cf-Connecting-Ip":["106.69.185.51"],"Cf-Ipcountry":["AU"],"Cf-Ray":["64d396e34aea1aac-SIN"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["09f81ca21200001aacac959000000001"],"Connection":["Keep-Alive"],"Cache-Control":["no-cache"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005877493,"status":502,"err_id":"0vkbgp8eh","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}

Backend Caddy log extract

{"level":"info","ts":"2021-05-10T21:41:50.878+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:50.879+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:50.887+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003347e0"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654111.0122516,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654111.0166075,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002f2700"}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-10T21:41:51.023+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-10T21:41:51.045+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=91239) - Caddy is running in the background