Also, if I move the handle block below further up above other handle blocks, or, happen to add a new handle block after this one, would I need to change the handle directive to route so that if there’s a match, it runs through to the next handle block?
…or, because I’m using reverse_proxy, which is terminal, to avoid any ambiguity, would this work?
If you unwrap those, then you lose the benefits of handle's mutual exclusivity and directive sorting order (which is guaranteed to be in the order they are written in the Caddyfile in relation to eachother) and the directives like respond and reverse_proxy would get sorted according to their directive order. The effect of this would be that respond @unknown "Denied" 403 would happen after the redir and header and basicauth which is probably not correct (doesn’t make sense to trigger auth when you’re going to respond with a 403 for example).
That said, I realized I made a mistake, because route is ordered after handle, so all the route blocks will get put at the end (you can see this in the adapt output).
The dumb/easy fix it to just wrap the whole thing in a single top-level route to tell the Caddyfile adapter “no, do it in this order” but that’s not so nice but in that case you don’t need some of the handle anymore because you can safely place directives in your Caddyfile in the right order by hand.
Ultimately, you’re writing a super complex config here, so it’s pretty tricky to get right (cause you have multiple paths through which a single request could go because you have multiple variables at play).
Ah hah…this is the routing first approach @matt refers to in said wiki article! @francislavoie this is all good stuff as it’s helping me cement my understanding.
… but logical and straightforward enough that mere mortals like myself can follow.
I know I’m pushing the envelope with Caddyfile, and I haven’t (yet, but could be coming soon!) hit any limitations using it, but I just love its simplicity with all the complexity handled under the hood.
I’ll test this out and report back later this evening (local time). Thank you for your help getting me to this point.
There are those that are less than in love with the Caddyfile, but the truth is there are very few things it can’t do.
The place where it can become onerous is when the complexity of a given configuration (such as yours) starts to expose nuances in the abstraction. Specifically, we begin to require a high level of understanding of how and why the Caddyfile maps out into Caddy’s “real” config (JSON) in order to write it properly, at which point some see it as more pragmatic to simply write directly in JSON.
But, in truth, the Caddyfile can handle this, which is pretty cool.
The top-level route wrapper worked a treat. Adopting a routing-first approach was the way to go in this case. Interestingly, all the handle blocks/wrapping have disappeared from my Caddyfile making it even more streamlined than it was before, while improving its overall readability as well.
Time to tinker with the backend.
EDIT: I had to remove the quotes from the header line…
Leaving them in resulted in Nextcloud complaining about Strict Transport Security and, curiously, also broke the connection between Nextcloud and OnlyOffice.
I just prefer to include the quotes around header values in situation where it could eventually have spaces in the value, because if you have a 3rd parameter (i.e. third Caddyfile token, separated by spaces), it becomes a header replacement operation which is rarely the intended effect.
Where to from here?
Here’s an extract of the frontend Caddy log post-restart.
{"level":"info","ts":"2021-05-09T22:55:21.393+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-09T22:55:21.393+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-09T22:55:21.950+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000344930"}
{"level":"debug","ts":"2021-05-09T22:55:21.950+0800","logger":"http.handlers.acme_server","msg":"unloading unused CA database","db_key":"local"}
{"level":"info","ts":"2021-05-09T22:55:21.952+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-09T22:55:21.952+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620572122.0956097,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620572122.1175373,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-09T22:55:22.125+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-09T22:55:22.126+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00033e850"}
{"level":"info","ts":"2021-05-09T22:55:22.155+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-09T22:55:22.155+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/09 22:55:35 Note: NSS support is not available on your platform
2021/05/09 22:55:35 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-09T22:55:35.176+0800","logger":"pki.ca.local","msg":"failed to install root
certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-09T22:55:35.176+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-09T22:55:35.176+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","*.udance.com.au","acme.lan","caffigoalkeeping.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","www.caffigoalkeeping.com.au","readymcgetty.com.au","www.readymcgetty.com.au","xenografix.com.au","udance.com.au","www.udance.com.au"]}
{"level":"warn","ts":"2021-05-09T22:55:35.191+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-09T22:55:35.191+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=77501) - Caddy is running in the background
…and an extract from the backend Caddy log post-restart:
{"level":"info","ts":"2021-05-09T22:55:57.526+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-09T22:55:57.526+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-09T22:55:57.529+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002c6d90"}
{"level":"info","ts":"2021-05-09T22:55:57.531+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-09T22:55:57.531+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620572157.6563253,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620572157.6602314,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-09T22:55:57.662+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-09T22:55:57.663+0800","logger":"tls.cache.maintenance","msg":"started backgroundcertificate maintenance","cache":"0xc0002de380"}
{"level":"info","ts":"2021-05-09T22:55:57.663+0800","logger":"http","msg":"server is listening only on theHTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-09T22:55:57.664+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-09T22:55:57.665+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-09T22:55:57.665+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-09T22:55:57.667+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-09T22:55:57.688+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-09T22:55:57.689+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-09T22:55:57.689+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=80607) - Caddy is running in the background
There’s an error about OCSP stapling in both Caddy logs, but I’m not sure what this means. Help! Please!
root@caddy:/.local/share/caddy/pki/authorities/local # ls -l
total 26
-rw------- 1 root wheel 680 May 8 04:00 intermediate.crt
-rw------- 1 root wheel 227 May 8 04:00 intermediate.key
-rw------- 1 root wheel 627 May 2 13:35 root.crt
-rw------- 1 root wheel 227 May 2 13:35 root.key
I wonder if it has something to do with the acme server and the FreeBSD platform. The message first appears when I enable the acme server. See post #2 in this thread.
There must be more logs after that for both of them, i.e. doing the ACME on the backend for that domain. That looks like you only got the logs from the restart and that’s it. Need to see the logs after it started running too, because some of this stuff doesn’t happen right in the startup phase, but after it.
Note sure why there’s a newline in there, but anyways I don’t think this is a problem, this just means Caddy wasn’t able to install the root certificate into the trust store of that system which might be fine.
But you could run caddy trust manually to make sure it was done right anyways (make sure to run that command such that Caddy knows that its storage location is /var/lib/caddy, i.e. the $HOME of the caddy user when running as a service). Caddy needs sudo/root access to install that which is the trouble.
Ignore that, not a problem. It’s just a warning. OCSP is not required, but strongly recommended, but isn’t really viable for mTLS. Basically it’s supposed to be a URL that Caddy can send a request to, to see if the certificate is still valid (i.e. hasn’t been revoked for any number of reasons). It’s decently important but public trust but for private it doesn’t matter so much.
My bad. That’s me breaking up long lines to copy across and, in this instance, forgetting to restore the line.
Good to know.
The first three related messages post-restart from the frontend log. There are a few more messages similar to this that follow, and then what appears to be a cert refresh, and then more error messages much like the first three.
...
{"level":"error","ts":"2021-05-09T23:00:37.235+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.21:22832","proto":"HTTP/2.0","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"Cf-Ray":["64cbcc60ddf29b97-DFW"],"X-Forwarded-Proto":["https"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cf-Request-Id":["09f33e108b00009b979e2cf000000001"],"Cdn-Loop":["cloudflare"],"Cf-Ipcountry":["US"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["192.0.91.177"],"Cf-Visitor":["{\"scheme\":\"https\"}"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006550785,"status":502,"err_id":"kx7sdf71h","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...
{"level":"error","ts":"2021-05-09T23:06:06.006+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"10.1.1.54:21726","proto":"HTTP/1.1","method":"POST","host":"test.udance.com.au","uri":"/wp-admin/admin-ajax.php?action=updraft_central","headers":{"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-admin/admin-ajax.php?action=updraft_central"],"Connection":["close"],"Content-Length":["1146"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.1; class-udrpc.php/1.4.21; https://blog.udance.com.au"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006877555,"status":502,"err_id":"jkzpr9x4p","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-09T23:07:06.187+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.118:47290","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Connection":["Keep-Alive"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Request-Id":["09f343ffe800000f3e8815d000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["64cbd5dfde3a0f3e-DFW"],"Cf-Connecting-Ip":["192.0.91.177"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006013134,"status":502,"err_id":"xv3yykkdr","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...
{"level":"info","ts":"2021-05-10T00:15:22.128+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["acme.lan"],"remaining":14096.871491679}
{"level":"info","ts":"2021-05-10T00:15:22.128+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["acme.lan"],"remaining":14096.871356348}
{"level":"info","ts":"2021-05-10T00:15:22.129+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.134+0800","logger":"tls.renew","msg":"lock acquired","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.135+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"acme.lan","remaining":14096.864875394}
{"level":"info","ts":"2021-05-10T00:15:22.137+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.137+0800","logger":"tls.renew","msg":"releasing lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-10T00:15:22.138+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["acme.lan"]}
{"level":"warn","ts":"2021-05-10T00:15:22.139+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T00:15:22.139+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["acme.lan"],"new_expiration":"2021-05-10T04:15:22.000Z"}
{"level":"error","ts":"2021-05-10T00:18:32.293+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.6:43830","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"Connection":["Keep-Alive"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["64cc3e840a3a2f25-DFW"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cf-Request-Id":["09f385668500002f25a01dc000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006488916,"status":502,"err_id":"pvzr5v6v0","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
...
Where there’s an error in the log, I notice this at the end of the line
I think this is the relevant bit. So like I said earlier, I think you need to run caddy trust on the front-end. Basically the backend did get certificates signed by the front-end, but then when the front-end goes to proxy, it doesn’t trust the backend’s cert because the root CA cert isn’t in the front-end’s system trust store (which is used when making HTTPS requests with the proxy).
Alternatively you can configure transport http with the tls_trusted_ca_certs with the path to the root CA cert file (i.e. /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt), but that’s a bit of a dumb workaround. Running caddy trust is the better option.
The caddy trust command is not very intelligent right now… the trouble is the user it runs as. If you just run it with caddy trustas your own user then it’ll try to use your user’s home. If you just run it with sudo caddy trust then it’ll use the root user’s home (i.e. /root). But what we actually want is the home of the caddy user (under which the systemd service runs Caddy) which is /var/lib/caddy. Hence hacking it by overriding HOME in that command
This made me realise I’d copied over the incorrect certificate to the backend Caddy server. I had copied across /.local/share/caddy/pki/authorities/local/root.crt instead of /var/lib/caddy/.local/share/caddy/pki/authorities/local/root.crt.
@Rob789 The above paragraph in your wiki article under the heading Backends should be updated to use the absolute path to avoid any ambiguity:
root@caddy:~ # sudo HOME=/var/lib/caddy caddy trust
2021/05/10 13:40:41.376 WARN ca.local installing root certificate (you might be prompted for password) {"path": "storage:pki/authorities/local/root.crt"}
2021/05/10 21:40:41 Note: NSS support is not available on your platform
2021/05/10 21:40:41 define JAVA_HOME environment variable to use the Java trust
trust: trust not supported
After copying across the correct cert to the backend server, I then restarted both frontend and backend Caddy services and attempted to access the test site. Sadly still no go
Frontend caddy log extract
{"level":"info","ts":"2021-05-10T21:41:44.437+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:44.437+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:44.443+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00035a690"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:44.444+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654104.5705228,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654104.6052015,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:44.613+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-10T21:41:44.614+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00032ff80"}
{"level":"info","ts":"2021-05-10T21:41:44.644+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:44.645+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-10T21:41:57.225+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/10 21:41:57 Note: NSS support is not available on your platform
2021/05/10 21:41:57 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-10T21:41:57.226+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com","readymcgetty.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","udance.com.au","www.caffigoalkeeping.com.au","www.readymcgetty.com.au","xenografix.com.au","acme.lan"]}
{"level":"info","ts":"2021-05-10T21:41:57.226+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-10T21:41:57.258+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:57.259+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=96255) - Caddy is running in the background
{"level":"info","ts":"2021-05-10T21:41:57.260+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"error","ts":"2021-05-10T21:42:08.298+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.166.143:50592","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Accept-Encoding":["gzip"],"Cf-Ipcountry":["AU"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22"],"Cf-Request-Id":["09f81c940700001aaca3904000000001"],"Cf-Ray":["64d396ccd9631aac-SIN"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Connection":["Keep-Alive"],"X-Forwarded-For":["106.69.185.51"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Gpc":["1"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Referer":["https://test.udance.com.au/"],"Cf-Connecting-Ip":["106.69.185.51"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006602499,"status":502,"err_id":"nfeudwy0e","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
{"level":"error","ts":"2021-05-10T21:42:11.888+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"162.158.165.126:18316","proto":"HTTP/1.1","method":"GET","host":"test.udance.com.au","uri":"/favicon.ico","headers":{"Accept":["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Dest":["image"],"Referer":["https://test.udance.com.au/"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["106.69.185.51"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Pragma":["no-cache"],"Sec-Gpc":["1"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; __cfduid=d73e22d7aa4142037dc86a5d6da76202d1618656636; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; cf_ob_info=502:64d396ccd9631aac:SIN; cf_use_ob=0"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.93 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Cf-Connecting-Ip":["106.69.185.51"],"Cf-Ipcountry":["AU"],"Cf-Ray":["64d396e34aea1aac-SIN"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["09f81ca21200001aacac959000000001"],"Connection":["Keep-Alive"],"Cache-Control":["no-cache"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005877493,"status":502,"err_id":"0vkbgp8eh","err_trace":"reverseproxy.statusError (reverseproxy.go:850)"}
Backend Caddy log extract
{"level":"info","ts":"2021-05-10T21:41:50.878+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-10T21:41:50.879+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-10T21:41:50.887+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0003347e0"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-10T21:41:50.888+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620654111.0122516,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620654111.0166075,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-10T21:41:51.019+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002f2700"}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-10T21:41:51.020+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-10T21:41:51.021+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-10T21:41:51.023+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-10T21:41:51.045+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-10T21:41:51.045+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=91239) - Caddy is running in the background
Wait a minute… Do you have the ca-certificates package installed? Or maybe that’s stored in a different location on your system… You’re on a BSD right?
Edit: Well I’m going to guess that since you’re on FreeBSD, the smallstep trust install command doesn’t know what to do which is why we see “trust not supported”. I’ll need to dig into that to find out when they message is written.
From a bit of Googling, looks like FreeBSD now uses Mozilla’s trust store (aka NSS) as the system trust. This seems to be shipped as a single file that apps are meant to read from. Idk what to suggest, cause I’m not a BSD user.
Aha - yeah, the trust tool doesn’t seem to support FreeBSD.
If you could comment on the issue and help move it along, that’d be cool. If you can figure out the right procedure to install a custom root cert on FreeBSD and help them test it, changes there should eventually land in Caddy.
I will make a note but the absolute path depends on a few things. For example, in my Wiki setup I run caddy with caddy start in /root which makes the absolute path /root/.local/share/caddy/pki/authorities/local but when you use systemd the path would be (for at least Debian) /var/lib/caddy/.local/share/caddy/pki/authorities/local
Okay (after regaining some composure), I must confess to being bamboozled at this point as this is now starting to go over my head. I’m having some difficulty connecting the dots.I don’t pretend to understand any of this stuff and what follows is just a little detective work on my part to see if anything has slipped through the cracks.
The last entry for the above issue was dated 10 April 2020. Seven days later on 17 April 2020 step-certificates and step-cli were ported to FreeBSD. Does this imply anything about the trust tool and FreeBSD, or am I barking up the wrong tree? I’m hoping this ignites a spark of hope. Assuming it does, are step-certificates and step-cli package dependencies for Caddy, and, with them installed, it may be possible to skirt around the ‘trust not supported’ error?
(you can see this in the adapt output).
but in that case you don’t need some of the 
