mTLS under FreeBSD

Rob789 · May 18, 2021, 6:50am

How I understood the Caddy’s local CA implementation is that it is using the Smallstep library. Thus, defining tls internal automatically makes use of the Smallstep CA. I don’t think Caddy has it’s own implementation for a certificate authority.

You can find more info on Github here.

basil · May 18, 2021, 6:59am

Hi @Rob789,

I should have been clearer … Caddy can use its own internal CA (based on the Smallstep library) or use a local step-ca instance…

Here’s an extract from the Smallstep reference I linked to…

Caddy comes with its own ACME server and by default it will generate an internal CA and issue certificates to itself. But, you can configure Caddy to use a local step-ca instance to obtain certificates.

Thanks for the Gihub link though. Makes for interesting reading.

francislavoie · May 18, 2021, 10:38am

If that’s possible, I’m not aware of it. But it would only be via JSON if it is. The facilities for the pki app are pretty limited from the Caddyfile.

/cc @matt

basil · May 18, 2021, 9:36pm

@francislavoie I’ve come full circle back to issue you identified in post #38 Bypassing the Caddy internal CA and using the Smallstep CA instead produces the same error. This Smallstep discussion has further detail.

So, for the moment, my mTLS journey has stalled . Until Smallstep fully support FreeBSD, mTLS via Caddy isn’t available for the FreeBSD platform.

francislavoie · May 18, 2021, 9:49pm

Like I said earlier, you don’t have to use caddy trust for it to work, you just need to install the root CA certificate whatever way the operating system expects you to do it. You have direct access to the CA cert, so you can copy it wherever it needs to be. The caddy trust command is just convenience to automate that process.

I don’t know enough about FreeBSD to know the right approach to actually install the cert into the system trust, but again, Google is your friend. I’m sure you can figure it out

And once you have figured out the process, you can comment on the issue on the smallstep repo to help them add support for FreeBSD. Then everyone wins.

basil · May 19, 2021, 9:14am

Well, that was the kick in the behind I needed to keep going for a bit longer, though I do have to say I’ve been out of my comfort zone for a while now. After trying a whole bunch of stuff that led nowhere, I might have made a little progress now. It turns out there are two issues I was grappling with:

The trust store issue
The ‘real’ root.crt location.

It turns out the second issue had a bearing on the first.

root.crt location

I was beating my head against the wall installing the root cert under /var/lib/caddy… into the trust store and not making any headway with mTLS.

The clue here was this note from @Rob789’s wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts :

NOTE: The absolute path for the certificate depends on how you run Caddy. Typically when you use systemd the absolute path will be /var/lib/caddy/.local/share/caddy/pki/authorities/local but if you run caddy with caddy start the path .local/share/caddy/pki/authorities/local

@Rob789 I’m curious to know where you came across this gem? Had I heeded the advice, my head might not have been as battered and bloodied

This is what made me change tack. When I looked more closely at both locations this is what I saw:

root@caddy:~ # ls -l /.local/share/caddy/pki/authorities/local/
total 26
-rw-------  1 root  wheel  676 May 19 12:27 intermediate.crt
-rw-------  1 root  wheel  227 May 19 12:27 intermediate.key
-rw-------  1 root  wheel  627 May  2 13:35 root.crt
-rw-------  1 root  wheel  227 May  2 13:35 root.key

root@caddy:~ # ls -l /var/lib/caddy/.local/share/caddy/pki/authorities/local/
total 26
-rw-------  1 root  wheel  676 May 10 11:06 intermediate.crt
-rw-------  1 root  wheel  227 May 10 11:06 intermediate.key
-rw-------  1 root  wheel  627 May 10 11:06 root.crt
-rw-------  1 root  wheel  227 May 10 11:06 root.key

There was a more recent date for the intermediate cert in the former, which led me to suspect that I needed to install the root cert from that directory in the trust store. As caddy start is used to run caddy in the environment, based on the note, this reinforced which root.crt I needed to install in the trust store. Once I understood this, I began to make some progress. mTLS is still not working completely though, but I’ll get to that shortly.

Trust root CA cert

So, I found a couple of resources that seemed to be pertinent.

How to install a private CA certificate on FreeBSD, and more recently…
Howto install a private CA certificate? - a FreeBSD forum thread.

Combining elements of both, I think I have installed the correct cert in the system trust. Here’s a log of the steps I took:

root@caddy:~ # freebsd-version
12.2-RELEASE-p6
root@caddy:~ # cp /.local/share/caddy/pki/authorities/local/root.crt /etc/ssl/certs
root@caddy:~ # openssl rehash /etc/ssl/certs
root@caddy:~ # openssl s_client -connect acme.lan:443 | grep -i -e verify
depth=1 CN = Caddy Local Authority - ECC Intermediate
verify return:1
depth=0
verify return:1
Verify return code: 0 (ok)

    Verify return code: 0 (ok)
root@caddy:~ #

Restarting the frontend Caddy service and checking the console log, there’s still the trust store error, but I assume that’s to be expected.

root@caddy:~ # tail --lines=20 /var/log/caddy.log
{"level":"info","ts":"2021-05-19T16:15:55.047+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00033a7e0"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:15:55.048+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412155.1001155,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412155.1057749,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-19T16:15:55.110+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000294690"}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:15:55.121+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"warn","ts":"2021-05-19T16:16:01.484+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/19 16:16:01 Note: NSS support is not available on your platform
2021/05/19 16:16:01 define JAVA_HOME environment variable to use the Java trust
{"level":"error","ts":"2021-05-19T16:16:01.485+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","readymcgetty.com.au","xenografix.com.au","www.caffigoalkeeping.com","acme.lan","www.udance.com.au","udance.com.au","www.caffigoalkeeping.com.au","caffigoalkeeping.com.au","*.udance.com.au","caffigoalkeeping.com","www.readymcgetty.com.au"]}
{"level":"info","ts":"2021-05-19T16:16:01.485+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-19T16:16:01.495+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:16:01.495+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=53177) - Caddy is running in the background
{"level":"info","ts":"2021-05-19T16:16:01.496+0800","logger":"tls","msg":"finished cleaning storage units"}

Restarting the backend Caddy service and checking the log, mTLS appears to be stirring into life!

{"level":"info","ts":"2021-05-19T16:22:28.408+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-19T16:22:28.408+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-19T16:22:28.411+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002e3110"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-19T16:22:28.413+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621412548.539171,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621412548.543643,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-19T16:22:28.546+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ec690"}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-19T16:22:28.547+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-19T16:22:28.548+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-19T16:22:28.550+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-19T16:22:28.572+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-19T16:22:28.572+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=28161) - Caddy is running in the background

Attempting to access test.udance.com.au, I see a 502 error in the browser, nothing in the backend Caddy log, but this error in the frontend log.

{"level":"error","ts":"2021-05-19T16:44:22.869+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"10.1.1.222:49381","proto":"HTTP/2.0","method":"GET","host":"test.udance.com.au","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Cookie":["tk_or=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wordpress_logged_in_6f3be6329744d07b768d1565b796af6d=basil%7C1645523796%7Cp7aCGGrrNSGs9CR4PRXTX8cgzxAPWRW5H3EMiHsK5v1%7Ca678f9423db6329357457c5b072e5840131c33f05e2c861be9d697954217ae95; wp-settings-1=libraryContent%3Dbrowse%26editor%3Dtinymce%26hidetb%3D1%26editor_plain_text_paste_warning%3D1; wp-settings-time-1=1613987799; tk_lr=%22https%3A%2F%2Fheimdall.udance.com.au%2F%22; wfwaf-authcookie-ce4a5970b2fd99774e0154e5e6b9716d=1%7Cadministrator%7Cd4b9b9e4e28bbee1e82f1b7f2b053da5fb2e984b668ed5b81d82bebf717d7436"],"Sec-Gpc":["1"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.005157315,"status":502,"err_id":"x10a03jus","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}

Have I installed the correct root cert in the system trust? I’m still not entirely sure. Can we get past this next stumbling block? That remains to be seen.

Relevant sections of the frontend Caddyfile…

...
# ACME server
acme.lan {
  acme_server
  tls internal
}
...
*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
  ...
    test         test.lan:443    yes    yes   yes        # test.udance.com.au
  ...
  route {
    ...
# Secure backend communication
    @mtls expression `{mtls} == "yes"`
    reverse_proxy @mtls {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
    ...
  }

Relevant sections of the backend Caddyfile (full copy in post #29) …

{
  ...
  acme_ca https://acme.lan/acme/local/directory
  acme_ca_root /etc/ssl/certs/root.crt
}

test.lan {
  ...
  }

Oh, once I worked out which root.crt to use, this appeared earlier in the day in the backend Caddy log. Seems promising.

{"level":"info","ts":"2021-05-19T14:39:19.704+0800","logger":"tls.renew","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:19.707+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"test.lan","remaining":-736110.707918771}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:19.713+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-19T14:39:20.069+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-19T14:39:20.138+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:44929","distributed":false}
{"level":"info","ts":"2021-05-19T14:39:20.492+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/BpA9sI3WDWa6fMtGGZufr5pRjKRKwo2l"}
{"level":"info","ts":"2021-05-19T14:39:20.704+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/gxvym5UeCVGotOR1rKJymjEj6AmDCq5X"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.705+0800","logger":"tls.renew","msg":"releasing lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-19T14:39:20.706+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["test.lan"]}
{"level":"warn","ts":"2021-05-19T14:39:20.707+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-19T14:39:20.707+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["test.lan"],"new_expiration":"2021-05-19T18:39:20.000Z"}

francislavoie · May 19, 2021, 2:53pm

It’s essentially the HOME environment variable of the user under which you ran Caddy.

Yeah, that’s harmless. Ignore it. In a later version, we’ll have an option to skip the install step which it tries at startup.

Did you also copy that newly found root cert to your backend Caddy instance too? It also needs that one to trust the frontend.

You might need to wipe the backend instance’s data storage to force it to fetch a new cert from the frontend with the right trust established maybe.

Yeah that looks pretty good.

Maybe there’s another step you need to take to install the root cert, cause Caddy doesn’t use openssl - it uses the implementation of crypto from the Go standard library.

I feel like I read in some cursory Googling that FreeBSD uses Mozilla’s NSS tools as the trust store. Maybe there’s something else to be done there on the frontend.

Or, you could configure reverse_proxy’s http transport with tls_trusted_ca_certs instead of mucking with the trust store. Less nice if a solution though.

basil · May 19, 2021, 3:21pm

Yes, I did.

Can you elaborate, please? I’m not exactly sure what’s involved.

I’ve mucked around so much with Caddy and Smallstep in the same space, I might have messed something up. I think I’m going to start with a fresh Caddy instance and see if I end up in the same place. This will give me the opportunity to upgrade to Caddy 2.4.0. With a bit of luck, it may just be something I’ve inadvertently broken. If the issue is reproducible, I’ll then reflect on this advice.

francislavoie · May 19, 2021, 3:30pm

Basically just rm -rf the contents of Caddy’s storage, then restart Caddy. But doing a fresh install avoids needing to do that

basil · May 19, 2021, 9:11pm

I rebuilt the frontend Caddy instance, but still end up with the ‘x509: certificate signed by unknown authority’ error…

root@caddy:~ # caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

root@caddy:~ # tail --lines=1 /var/log/caddy.log
{"level":"error","ts":"2021-05-20T04:46:11.333+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.21:60838","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["65202c54dc6dc81e-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["0a27fa09070000c81e109a5000000001"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006129711,"status":502,"err_id":"ghqnryw6i","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}

I deleted the backend Caddy storage, restarted Caddy and the log showed that a new cert was fetched from the frontend. Would this behaviour suggest that the root cert was correctly inserted in the system trust in the frontend Caddy instance?

{"level":"info","ts":"2021-05-20T04:24:26.146+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-20T04:24:26.147+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-20T04:24:26.149+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002bafc0"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621455866.2774534,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621455866.2815413,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002e48c0"}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"serving initial configuration"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls.obtain","msg":"acquiring lock","identifier":"test.lan"}
Successfully started Caddy (pid=39197) - Caddy is running in the background
{"level":"info","ts":"2021-05-20T04:24:26.309+0800","logger":"tls.obtain","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:27.120+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"tls-alpn-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-20T04:24:27.167+0800","logger":"tls","msg":"served key authentication certificate","server_name":"test.lan","challenge":"tls-alpn-01","remote":"10.1.1.4:26438","distributed":false}
{"level":"info","ts":"2021-05-20T04:24:27.666+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/m9Oisrd24B6ktWgqfg1lxbLamzzuEFY3"}
{"level":"info","ts":"2021-05-20T04:24:27.957+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/i9QC6UPnO0BdLB2U1QP1LSmhL0WO5OA1"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"releasing lock","identifier":"test.lan"}
{"level":"warn","ts":"2021-05-20T04:24:27.960+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}

If backend behaviour suggests that the cert was correctly installed in the frontend system trust, is it time to comment in the Smallstep repo?

ca_root_nss is one of the installed packages.

root@caddy:~ # pkg info
bash-5.1.4_1                   GNU Project's Bourne Again SHell
ca_root_nss-3.63               Root certificate bundle from the Mozilla Project
curl-7.76.0                    Command line tool and library for transferring data with URLs
cvsps-2.1_2                    Create patchset information from CVS
expat-2.2.10                   XML 1.0 parser written in C
gettext-runtime-0.21           GNU gettext runtime libraries and programs
git-2.31.1_1                   Distributed source code management tool
go-1.16.4,1                    Go programming language
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libnghttp2-1.43.0              HTTP/2.0 C Library
nano-5.5                       Nano's ANOther editor, an enhanced free Pico clone
p5-Authen-SASL-2.16_1          Perl5 module for SASL authentication
p5-CGI-4.51                    Handle Common Gateway Interface requests and responses
p5-Clone-0.45                  Recursively copy Perl datatypes
p5-Digest-HMAC-1.03_1          Perl5 interface to HMAC Message-Digest Algorithms
p5-Encode-Locale-1.05          Determine the locale encoding
p5-Error-0.17029               Error/exception handling in object-oriented programming style
p5-GSSAPI-0.28_1               Perl extension providing access to the GSSAPIv2 library
p5-HTML-Parser-3.75            Perl5 module for parsing HTML documents
p5-HTML-Tagset-3.20_1          Some useful data table in parsing HTML
p5-HTTP-Date-6.05              Conversion routines for the HTTP protocol date formats
p5-HTTP-Message-6.28           Representation of HTTP style messages
p5-IO-HTML-1.001_1             Open an HTML file with automatic charset detection
p5-IO-Socket-INET6-2.72_1      Perl module with object interface to AF_INET6 domain sockets
p5-IO-Socket-SSL-2.070         Perl5 interface to SSL sockets
p5-LWP-MediaTypes-6.04         Guess media type for a file or a URL
p5-Mozilla-CA-20200520         Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-SSLeay-1.88             Perl5 interface to SSL
p5-Socket6-0.29                IPv6 related part of the C socket.h defines and structure manipulators
p5-TimeDate-2.33,1             Perl5 module containing a better/faster date parser for absolute dates
p5-URI-5.07                    Perl5 interface to Uniform Resource Identifier (URI) references
pcre2-10.36                    Perl Compatible Regular Expressions library, version 2
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.16.3                     Package manager
python37-3.7.10                Interpreted object-oriented programming language
readline-8.1.0                 Library for editing command lines as they are typed

Is it time to consider this, or exhaust other options first e.g. try to understand and resolve the observed frontend error?

When comparing the frontend and backend Caddy logs, it’s pretty clear that the frontend is not communicating with the backend when the error occurs.

francislavoie · May 19, 2021, 9:33pm

No, that just means that you used the right certificate for acme_ca_root on the backend. The frontend’s system trust is not involved when actually issuing certificates to other Caddy instances. It’s involved when actually proxying requests to the other instances.

It’s time to comment there as soon as you have something useful to report to them in terms of how best to install root CA certs on FreeBSD.

Good to know. So you’ll need to figure out how to add your own certs to that bundle.

I mean, you can try it at any point. If it works, then you know that it’s an option.

Right, cause the TLS handshake fails before the backend decides to log anything about the request. If you enable the debug global option, you might see some logs on the backend though.

basil · May 20, 2021, 6:15am

Well, 16 days later and we have liftoff! …sort of

The clues that led me to a ‘solution’…
I looked a little deeper into ca_root_nss:

root@caddy:~ # pkg info -l ca_root_nss
ca_root_nss-3.63:
        /etc/ssl/cert.pem
        /usr/local/etc/ssl/cert.pem.sample
        /usr/local/openssl/cert.pem.sample
        /usr/local/share/certs/ca-root-nss.crt
        /usr/local/share/licenses/ca_root_nss-3.63/LICENSE
        /usr/local/share/licenses/ca_root_nss-3.63/MPL20
        /usr/local/share/licenses/ca_root_nss-3.63/catalog.mk

…and then appended root.crt to ca-root-nss.crt:

root@caddy:~ # cat /.local/share/caddy/pki/authorities/local/root.crt >> /usr/local/share/certs/ca-root-nss.crt

I then restarted frontend Caddy and noticed a change in the log regarding the trust of the root cert:

{"level":"info","ts":"2021-05-20T13:13:14.114+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-20T13:13:14.114+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-20T13:13:14.117+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000315d50"}
{"level":"info","ts":"2021-05-20T13:13:14.118+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T13:13:14.118+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621487594.170676,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621487594.1781077,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-20T13:13:14.181+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-20T13:13:14.181+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ce5b0"}
{"level":"info","ts":"2021-05-20T13:13:14.191+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:13:14.191+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["xenografix.com.au","acme.lan","www.readymcgetty.com.au","www.caffigoalkeeping.com.au","readymcgetty.com.au","www.udance.com.au","udance.com.au","www.xenografix.com.au","caffigoalkeeping.com","*.udance.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com.au"]}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-20T13:13:20.994+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:13:20.997+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-20T13:13:21.001+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:13:21.001+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=64612) - Caddy is running in the background

Accessing the test site… Nice!

However, I don’t see anything in the frontend or backend Caddy logs unless I turn debug on. It would be nice to have some basic logging happening for mTLS.

The system trust solution I stumbled upon is semi-permanent at best. The clue was in this FreeBSD bug report Bug 160387 - security/ca_root_nss: Allow user to trust extra local certificates. The trust breaks if the ca_root_nss package is updated. To fix it, it’s necessary to append root.crt to ca-root-nss.crt again. For reasons described in the bug report, a permanent solution is unlikely to be forthcoming.

Question: How do I convince myself that the communication path between the backend and frontend Caddy services is actually encrypted?

francislavoie · May 20, 2021, 6:32am

The reverse_proxy logs should show that the request has TLS on it. If it does, then you’re good to go. Not much else to say, really. There’s nothing inherent about mTLS that can be logged, it’s just regular HTTPS proxying, plus non-public trust. That’s all.

Figures. That’s kinda annoying. There might be a way to hook into your package manager to run some script after each time a package is updated… maybe? I know nothing of FreeBSD’s package manager.

But you could report this on the smallstep repo. Should be helpful.

basil · May 20, 2021, 4:00pm

The euphoria was somewhat short-lived. I’m no longer able to access the test site.

In the few hours I’ve been away, this is what’s accumulated in the logs:

Backend Caddy log:

{"level":"info","ts":"2021-05-20T13:39:29.599+0800","logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:52184","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1684"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":"2021-05-20T13:39:29.600+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-20T13:39:29.601+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:39:29.601+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:39:29.603+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"warn","ts":"2021-05-20T13:39:29.604+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:39:29.604+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ca700"}
{"level":"info","ts":"2021-05-20T13:39:30.662+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002c5570"}
{"level":"info","ts":"2021-05-20T13:39:30.663+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:39:30.663+0800","logger":"admin.api","msg":"load complete"}
{"level":"info","ts":"2021-05-20T13:39:31.208+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T20:39:29.606+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["test.lan"],"remaining":14017.393058953}
{"level":"info","ts":"2021-05-20T20:39:29.607+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["test.lan"],"remaining":14017.392423428}
{"level":"info","ts":"2021-05-20T20:39:29.608+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:29.640+0800","logger":"tls.renew","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:29.640+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"test.lan","remaining":14017.359107754}
{"level":"info","ts":"2021-05-20T20:39:29.642+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T20:39:29.642+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T20:39:29.998+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-20T20:39:30.055+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:47482","distributed":false}
{"level":"info","ts":"2021-05-20T20:39:30.404+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri"}
{"level":"info","ts":"2021-05-20T20:39:30.583+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/fPbBOhtqkDfWJOdMMvB2B6kKNVTeAvAQ"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls.renew","msg":"releasing lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["test.lan"]}
{"level":"warn","ts":"2021-05-20T20:39:30.586+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T20:39:30.586+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["test.lan"],"new_expiration":"2021-05-21T00:39:30.000Z"}

Frontend Caddy log - noticeable events.

{"level":"info","ts":"2021-05-20T13:39:55.408+0800","logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:27757","headers":{"Accept-Encoding":["gzip"],"Content-Length":["10520"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":"2021-05-20T13:39:55.412+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-20T13:39:55.412+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003060e0"}
{"level":"info","ts":"2021-05-20T13:39:55.413+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:39:55.413+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:40:01.692+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.caffigoalkeeping.com.au","udance.com.au","www.readymcgetty.com.au","www.xenografix.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","readymcgetty.com.au","acme.lan","www.caffigoalkeeping.com","caffigoalkeeping.com","xenografix.com.au"]}
{"level":"warn","ts":"2021-05-20T13:40:01.699+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:40:01.699+0800","logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000178460"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","logger":"admin.api","msg":"load complete"}
{"level":"info","ts":"2021-05-20T13:40:02.344+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}

...

{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["acme.lan"],"remaining":14226.584475591}
{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["acme.lan"],"remaining":14226.584391609}
{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.471+0800","logger":"tls.renew","msg":"lock acquired","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.471+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"acme.lan","remaining":14226.528197201}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls.renew","msg":"releasing lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["acme.lan"]}
{"level":"warn","ts":"2021-05-20T19:49:55.474+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T19:49:55.474+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["acme.lan"],"new_expiration":"2021-05-20T23:49:55.000Z"}

...

2021/05/20 20:39:29 {"newNonce":"https://acme.lan/acme/local/new-nonce","newAccount":"https://acme.lan/acme/local/new-account","newOrder":"https://acme.lan/acme/local/new-order","revokeCert":"https://acme.lan/acme/local/revoke-cert","keyChange":"https://acme.lan/acme/local/key-change"}
2021/05/20 20:39:29 {"status":"pending","expires":"2021-05-21T12:39:30Z","identifiers":[{"type":"dns","value":"test.lan"}],"notBefore":"2021-05-20T12:39:30Z","notAfter":"2021-05-21T00:39:30Z","authorizations":["https://acme.lan/acme/local/authz/niNSkBvVy4bwYMeKQvS48pfBWzgw4DRY"],"finalize":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri/finalize"}
2021/05/20 20:39:29 {"identifier":{"type":"dns","value":"test.lan"},"status":"pending","expires":"2021-05-21T12:39:30Z","challenges":[{"type":"http-01","status":"pending","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"},{"type":"tls-alpn-01","status":"pending","token":"7tjZva3KtSzzdnGtK6YPOzNoasbiVcZW","url":"https://acme.lan/acme/local/challenge/yVYPnacjhH6QmzUhZpjNPOfTGPEQWHwV"},{"type":"dns-01","status":"pending","token":"CQzbgaNLQdMzir80pqYnQtX74U0tRNod","url":"https://acme.lan/acme/local/challenge/mBUvK1duKwkNj1VeQ3ysh4POvgKfqceU"}],"wildcard":false}
2021/05/20 20:39:30 {"type":"http-01","status":"valid","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","validated":"2021-05-20T12:39:30Z","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"}
2021/05/20 20:39:30 {"identifier":{"type":"dns","value":"test.lan"},"status":"valid","expires":"2021-05-21T12:39:30Z","challenges":[{"type":"http-01","status":"valid","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","validated":"2021-05-20T12:39:30Z","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"},{"type":"tls-alpn-01","status":"pending","token":"7tjZva3KtSzzdnGtK6YPOzNoasbiVcZW","url":"https://acme.lan/acme/local/challenge/yVYPnacjhH6QmzUhZpjNPOfTGPEQWHwV"},{"type":"dns-01","status":"pending","token":"CQzbgaNLQdMzir80pqYnQtX74U0tRNod","url":"https://acme.lan/acme/local/challenge/mBUvK1duKwkNj1VeQ3ysh4POvgKfqceU"}],"wildcard":false}
2021/05/20 20:39:30 {"status":"valid","expires":"2021-05-21T12:39:30Z","identifiers":[{"type":"dns","value":"test.lan"}],"notBefore":"2021-05-20T12:39:30Z","notAfter":"2021-05-21T00:39:30Z","authorizations":["https://acme.lan/acme/local/authz/niNSkBvVy4bwYMeKQvS48pfBWzgw4DRY"],"finalize":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri/finalize","certificate":"https://acme.lan/acme/local/certificate/fPbBOhtqkDfWJOdMMvB2B6kKNVTeAvAQ"}

...

There’s a pattern. In the frontend log, you can match up the first and third groups with what’s in the backend log. Interestingly, the format of the third group changes in the log, but then returns to normal in the messages that follow this group.I’m not sure if there’s any significance in this.

When I attempt to acces the site externally, I can a redirection from test.udance.com.au to test.lan.

What’s probably more telling is when I access test.udance.com.au from the local network (split DNS arrangement). There’s a privacy error and a message about an invalid CA.

francislavoie · May 20, 2021, 4:07pm

I’m not really gleaning anything useful from that. Make sure DNS is correct, make sure the root CA certificate is still in the NSS bundle (did it update since last night lol)?

The logs look pretty normal aside from that change in format, which I can’t explain.

basil · May 20, 2021, 4:30pm

It all checks out. The way I recovered was to clear the backend Caddy storage and restart Caddy so it would acquire a new certificate.

{"level":"info","ts":"2021-05-21T00:17:26.889+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-21T00:17:26.889+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-21T00:17:26.892+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000182770"}
{"level":"info","ts":"2021-05-21T00:17:26.893+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-21T00:17:26.894+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621527447.0243363,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621527447.0288177,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-21T00:17:27.032+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-21T00:17:27.033+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-21T00:17:27.033+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-21T00:17:27.033+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002eca10"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","msg":"serving initial configuration"}
{"level":"info","ts":"2021-05-21T00:17:27.036+0800","logger":"tls.obtain","msg":"acquiring lock","identifier":"test.lan"}
Successfully started Caddy (pid=21920) - Caddy is running in the background
{"level":"info","ts":"2021-05-21T00:17:27.072+0800","logger":"tls.obtain","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-21T00:17:27.425+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-21T00:17:27.425+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-21T00:17:27.681+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-21T00:17:27.728+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:23813","distributed":false}
{"level":"info","ts":"2021-05-21T00:17:28.093+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/Yvy89ZiAowVe5FrW4b1vZUEm7SGMamuH"}
{"level":"info","ts":"2021-05-21T00:17:28.306+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/5CiXEmiAu0u3oOf17Rdne0iK3ThKobb0"}
{"level":"info","ts":"2021-05-21T00:17:28.307+0800","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-21T00:17:28.307+0800","logger":"tls.obtain","msg":"releasing lock","identifier":"test.lan"}
{"level":"warn","ts":"2021-05-21T00:17:28.309+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}

I also had to clear browser caches as the browsers seemed to remember the redirection to test.lan. I’ll monitor this over the next few hours and see if the issue occurs again.

basil · May 21, 2021, 5:00am

The mTLS service I got working yesterday is extremely fragile. It collapsed a few minutes after I got it working last night and I haven’t been able to resurrect it since. So, a hard reset…

I rebuilt the Caddy front end.
I added the root.crt to the NSS bundle in the frontend and copied it to the backend as well.
I cleared Caddy storage in the backend.
I turned debug on in both the frontend and backend Caddyfiles.
I restarted both Caddy frontend and backend services.
I cleared my browser cache.

I then attempted to access the test site (test.udance.com.au) on the local network and immediately got directed to test.lan.

The local DNS resolves acme.lan → 10.1.1.4 and test.lan → 10.1.1.50. My local machine has an IP of 10.1.1.222.

This is what appeared in the backend Caddy log at the time I accessed the test site.

{"level":"debug","ts":"2021-05-21T11:55:22.637+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-For":["10.1.1.222"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-21T11:55:22.637+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_SEC_GPC":"1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"25859","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-21T11:55:24.285+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","headers":{"Accept-Encoding":["deflate, gzip"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Content-Length":["0"],"Accept":["*/*"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621569324.2422480583190917968750","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"25859","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-21T11:55:24.399+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","headers":{"Accept-Encoding":["deflate, gzip"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Content-Length":["0"],"Accept":["*/*"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"]},"status":200}
{"level":"debug","ts":"2021-05-21T11:55:24.428+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Gpc":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}
{"level":"debug","ts":"2021-05-21T11:55:24.494+0800","logger":"http.stdlib","msg":"http: TLS handshake error from 10.1.1.222:64614: remote error: tls: unknown certificate"}

…and in the frontend Caddy log…

{"level":"debug","ts":"2021-05-21T11:55:24.399+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:30648","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","headers":{"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750"],"X-Forwarded-Proto":["https"],"Content-Length":["0"],"X-Forwarded-Host":["test.udance.com.au"],"Accept-Encoding":["deflate, gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"],"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Fri, 21 May 2021 03:55:24 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-21T11:55:24.427+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:64602","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"Sec-Gpc":["1"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-For":["10.1.1.222"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-Host":["test.udance.com.au"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"heimdall.udance.com.au"}},"headers":{"Date":["Fri, 21 May 2021 03:55:24 GMT"],"Content-Type":["text/html; charset=UTF-8"],"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"X-Redirect-By":["WordPress"],"Content-Length":["0"]},"status":301}

From my reckoning, things seem to collapse starting with the last entry in the frontend log (time T11:55:24.427). There’s a reference to heimdall.udance.com.au (IP 10.1.1.23)…I’m not sure why? From that time forward it all seems to fall apart in the backend log (the last two entries).

Unrelated, but probably worth mentioning…When I rebuilt the Caddy frontend this morning using xcaddy including the Cloudflare module, Caddy got upgraded:

root@caddy:~ # caddy version
v2.4.1 h1:kAJ0JB5Xk5gPdTH/27S5cyoMGqD5lBAe9yZ8zTjVJa0=

On the backend, I use a Caddy static binary. Running caddy upgrade does not upgrade the binary.

root@wordpress:~ # caddy upgrade
2021/05/21 05:21:02.466 INFO    this executable will be replaced        {"path": "/usr/local/bin/caddy"}
2021/05/21 05:21:02.467 INFO    requesting build        {"os": "freebsd", "arch": "amd64", "packages": []}
2021/05/21 05:21:03.484 INFO    build acquired; backing up current executable   {"current_path": "/usr/local/bin/caddy", "backup_path": "/usr/local/bin/caddy.tmp"}
2021/05/21 05:21:03.485 INFO    downloading binary      {"source": "https://caddyserver.com/api/download?arch=amd64&os=freebsd", "destination": "/usr/local/bin/caddy"}
2021/05/21 05:21:16.055 INFO    download successful; displaying new binary details      {"location": "/usr/local/bin/caddy"}

Module versions:

admin.api.load v2.4.0
admin.api.metrics v2.4.0
admin.api.reverse_proxy v2.4.0
caddy.adapters.caddyfile v2.4.0
caddy.config_loaders.http v2.4.0
caddy.listeners.tls v2.4.0
caddy.logging.encoders.console v2.4.0
caddy.logging.encoders.filter v2.4.0
caddy.logging.encoders.filter.delete v2.4.0
caddy.logging.encoders.filter.ip_mask v2.4.0
caddy.logging.encoders.filter.replace v2.4.0
caddy.logging.encoders.json v2.4.0
caddy.logging.encoders.single_field v2.4.0
caddy.logging.writers.discard v2.4.0
caddy.logging.writers.file v2.4.0
caddy.logging.writers.net v2.4.0
caddy.logging.writers.stderr v2.4.0
caddy.logging.writers.stdout v2.4.0
caddy.storage.file_system v2.4.0
http v2.4.0
http.authentication.hashes.bcrypt v2.4.0
http.authentication.hashes.scrypt v2.4.0
http.authentication.providers.http_basic v2.4.0
http.encoders.gzip v2.4.0
http.encoders.zstd v2.4.0
http.handlers.acme_server v2.4.0
http.handlers.authentication v2.4.0
http.handlers.encode v2.4.0
http.handlers.error v2.4.0
http.handlers.file_server v2.4.0
http.handlers.headers v2.4.0
http.handlers.map v2.4.0
http.handlers.metrics v2.4.0
http.handlers.push v2.4.0
http.handlers.request_body v2.4.0
http.handlers.reverse_proxy v2.4.0
http.handlers.rewrite v2.4.0
http.handlers.static_response v2.4.0
http.handlers.subroute v2.4.0
http.handlers.templates v2.4.0
http.handlers.vars v2.4.0
http.matchers.expression v2.4.0
http.matchers.file v2.4.0
http.matchers.header v2.4.0
http.matchers.header_regexp v2.4.0
http.matchers.host v2.4.0
http.matchers.method v2.4.0
http.matchers.not v2.4.0
http.matchers.path v2.4.0
http.matchers.path_regexp v2.4.0
http.matchers.protocol v2.4.0
http.matchers.query v2.4.0
http.matchers.remote_ip v2.4.0
http.matchers.vars v2.4.0
http.matchers.vars_regexp v2.4.0
http.precompressed.br v2.4.0
http.precompressed.gzip v2.4.0
http.precompressed.zstd v2.4.0
http.reverse_proxy.selection_policies.cookie v2.4.0
http.reverse_proxy.selection_policies.first v2.4.0
http.reverse_proxy.selection_policies.header v2.4.0
http.reverse_proxy.selection_policies.ip_hash v2.4.0
http.reverse_proxy.selection_policies.least_conn v2.4.0
http.reverse_proxy.selection_policies.random v2.4.0
http.reverse_proxy.selection_policies.random_choose v2.4.0
http.reverse_proxy.selection_policies.round_robin v2.4.0
http.reverse_proxy.selection_policies.uri_hash v2.4.0
http.reverse_proxy.transport.fastcgi v2.4.0
http.reverse_proxy.transport.http v2.4.0
pki v2.4.0
tls v2.4.0
tls.certificates.automate v2.4.0
tls.certificates.load_files v2.4.0
tls.certificates.load_folders v2.4.0
tls.certificates.load_pem v2.4.0
tls.certificates.load_storage v2.4.0
tls.handshake_match.remote_ip v2.4.0
tls.handshake_match.sni v2.4.0
tls.issuance.acme v2.4.0
tls.issuance.internal v2.4.0
tls.issuance.zerossl v2.4.0
tls.stek.distributed v2.4.0
tls.stek.standard v2.4.0

  Standard modules: 83

  Non-standard modules: 0

  Unknown modules: 0

Version:
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

2021/05/21 05:21:16.218 INFO    upgrade successful; please restart any running Caddy instances  {"executable": "/usr/local/bin/caddy"}
root@wordpress:~ # service caddy restart
Stopping caddy.
Waiting for PIDS: 32125.
root@wordpress:~ # caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

basil · May 22, 2021, 4:23am

The next thing to try was to reset the local DNS resolver (DNSMasq on DD-WRT) just in case there was something weird happening in that space.

Attempting to access the test site test.udance.com.au externally:

This is what appeared in the logs:

Frontend:

{"level":"debug","ts":"2021-05-22T11:56:04.728+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:21924","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","headers":{"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-For":["10.1.1.50"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125"],"X-Forwarded-Proto":["https"],"Content-Length":["0"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"],"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Sat, 22 May 2021 03:56:04 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-22T11:56:04.757+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"172.70.49.149:43108","proto":"HTTP/1.1","method":"GET","host":"test.lan:443","uri":"/","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Cf-Ray":["65331cc398700a76-KIX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua-Mobile":["?1"],"Save-Data":["on"],"Cdn-Loop":["cloudflare"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"Cf-Ipcountry":["AU"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip"],"Cf-Connecting-Ip":["49.196.150.225"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Date":["Sat, 22 May 2021 03:56:04 GMT"],"Content-Type":["text/html; charset=UTF-8"],"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"X-Redirect-By":["WordPress"],"Content-Length":["0"]},"status":301}

Backend:

{"level":"debug","ts":"2021-05-22T11:56:02.905+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-Site":["none"],"Cf-Ray":["65331cc398700a76-KIX"],"Sec-Fetch-Mode":["navigate"],"Save-Data":["on"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Accept-Encoding":["gzip"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.150.225"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-22T11:56:02.905+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-Site":["none"],"Cf-Ray":["65331cc398700a76-KIX"],"Sec-Fetch-Mode":["navigate"],"Save-Data":["on"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.150.225"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"49.196.150.225","HTTP_CF_IPCOUNTRY":"AU","HTTP_CF_RAY":"65331cc398700a76-KIX","HTTP_CF_REQUEST_ID":"0a33d04e4300000a768e190000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_COOKIE":"tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22","HTTP_HOST":"test.lan:443","HTTP_SAVE_DATA":"on","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?1","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36","HTTP_X_FORWARDED_FOR":"49.196.150.225, 172.70.49.149, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33756","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T11:56:04.612+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","headers":{"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Content-Length":["0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621655764.5685970783233642578125","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33756","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T11:56:04.726+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","headers":{"Content-Length":["0"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"X-Powered-By":["PHP/7.4.16"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-22T11:56:04.756+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.150.225"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-Site":["none"],"Cf-Ray":["65331cc398700a76-KIX"],"Sec-Fetch-Mode":["navigate"],"Save-Data":["on"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}

Repeating the exercise, but accessing the site internally (split-DNS):

This is what appears in the logs:

Frontend:

{"level":"debug","ts":"2021-05-22T12:10:07.719+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:37639","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","headers":{"Content-Length":["0"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept-Encoding":["deflate, gzip"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Sat, 22 May 2021 04:10:07 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"]},"status":200}
{"level":"debug","ts":"2021-05-22T12:10:07.747+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:59220","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["test.udance.com.au"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["10.1.1.222"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Redirect-By":["WordPress"],"Content-Length":["0"],"Date":["Sat, 22 May 2021 04:10:07 GMT"],"Content-Type":["text/html; charset=UTF-8"],"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"]},"status":301}

Backend:

{"level":"debug","ts":"2021-05-22T12:10:05.992+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"X-Forwarded-For":["10.1.1.222"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-22T12:10:05.992+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33137","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T12:10:07.605+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Content-Length":["0"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept":["*/*"],"Accept-Encoding":["deflate, gzip"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621656607.5627350807189941406250","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33137","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T12:10:07.719+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","headers":{"Content-Length":["0"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept":["*/*"],"X-Forwarded-Host":["test.udance.com.au"],"Accept-Encoding":["deflate, gzip"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"X-Powered-By":["PHP/7.4.16"]},"status":200}
{"level":"debug","ts":"2021-05-22T12:10:07.747+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}
{"level":"debug","ts":"2021-05-22T12:10:07.792+0800","logger":"http.stdlib","msg":"http: TLS handshake error from 10.1.1.222:59223: remote error: tls: unknown certificate"}

EDIT: I believe why mTLS was so flaky when I first got it working was due to WordPress frontend cachiing on the test site. I’ve turned that off, but I’ve left WP backend object caching (Redis) on. Since then, I’m consistently getting the behaviour described in this post.

basil · May 22, 2021, 5:17am

Breaking news! I can access other sub-paths of the test site without issue. It’s the subdomain level that has an issue (previous post).

Externally:

Frontend log:

{"level":"debug","ts":"2021-05-22T12:49:27.298+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"162.158.119.150:31132","proto":"HTTP/1.1","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Accept-Encoding":["gzip"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?1"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Save-Data":["on"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cf-Ray":["65336afc8d281d87-NRT"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Mode":["navigate"],"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"],"Cf-Ipcountry":["AU"],"Cf-Connecting-Ip":["49.196.150.225"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Permitted-Cross-Domain-Policies":["none"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Set-Cookie":["phpMyAdmin_https=ret9asq4j2diuif6utq6udgtcr; path=/phpmyadmin/; secure; HttpOnly"],"Referrer-Policy":["no-referrer"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Robots-Tag":["noindex, nofollow"],"Date":["Sat, 22 May 2021 04:49:27 GMT"],"Content-Encoding":["gzip"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Last-Modified":["Sat, 22 May 2021 04:49:27 +0000"],"Server":["Caddy"],"X-Powered-By":["PHP/7.4.16"],"X-Xss-Protection":["1; mode=block"],"Content-Type":["text/html; charset=utf-8"],"Expires":["Sat, 22 May 2021 04:49:27 +0000"],"X-Ob_mode":["1"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Pragma":["no-cache"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"]},"status":200}

Backend log:

{"level":"debug","ts":"2021-05-22T12:49:27.050+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:27626","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["65336afc8d281d87-NRT"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Save-Data":["on"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Cf-Connecting-Ip":["49.196.150.225"],"Sec-Fetch-Mode":["navigate"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-22T12:49:27.051+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:27626","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["65336afc8d281d87-NRT"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Save-Data":["on"],"Cf-Connecting-Ip":["49.196.150.225"],"Sec-Fetch-Mode":["navigate"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"49.196.150.225","HTTP_CF_IPCOUNTRY":"AU","HTTP_CF_RAY":"65336afc8d281d87-NRT","HTTP_CF_REQUEST_ID":"0a340131d500001d87e5180000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F","HTTP_HOST":"test.lan:443","HTTP_SAVE_DATA":"on","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?1","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36","HTTP_X_FORWARDED_FOR":"49.196.150.225, 162.158.119.150, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"27626","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T12:49:27.300+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:27626","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["65336afc8d281d87-NRT"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Save-Data":["on"],"Cf-Connecting-Ip":["49.196.150.225"],"Sec-Fetch-Mode":["navigate"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Expires":["Sat, 22 May 2021 04:49:27 +0000"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Referrer-Policy":["no-referrer"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Vary":["Accept-Encoding"],"X-Powered-By":["PHP/7.4.16"],"Set-Cookie":["phpMyAdmin_https=ret9asq4j2diuif6utq6udgtcr; path=/phpmyadmin/; secure; HttpOnly"],"X-Robots-Tag":["noindex, nofollow"],"Content-Type":["text/html; charset=utf-8"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Xss-Protection":["1; mode=block"],"X-Content-Type-Options":["nosniff"],"Pragma":["no-cache"],"Last-Modified":["Sat, 22 May 2021 04:49:27 +0000"],"Content-Encoding":["gzip"],"X-Ob_mode":["1"],"X-Frame-Options":["DENY"]},"status":200}

Internally:

Frontend log:

{"level":"debug","ts":"2021-05-22T13:08:40.408+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:60356","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["10.1.1.222"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Purpose":["prefetch"],"X-Forwarded-Host":["test.udance.com.au"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Encoding":["gzip"],"Expires":["Sat, 22 May 2021 05:08:40 +0000"],"X-Frame-Options":["DENY"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Robots-Tag":["noindex, nofollow"],"X-Xss-Protection":["1; mode=block"],"Last-Modified":["Sat, 22 May 2021 05:08:40 +0000"],"Pragma":["no-cache"],"Server":["Caddy"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"],"X-Powered-By":["PHP/7.4.16"],"Date":["Sat, 22 May 2021 05:08:40 GMT"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Content-Type":["text/html; charset=utf-8"],"Referrer-Policy":["no-referrer"],"Set-Cookie":["phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e; path=/phpmyadmin/; secure; HttpOnly"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Ob_mode":["1"]},"status":200}
{"level":"debug","ts":"2021-05-22T13:08:41.540+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:60356","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"X-Forwarded-For":["10.1.1.222"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Ob_mode":["1"],"X-Robots-Tag":["noindex, nofollow"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Pragma":["no-cache"],"Referrer-Policy":["no-referrer"],"Set-Cookie":["phpMyAdmin_https=h65e8g79vlpcums8lpsjq0h9b7; path=/phpmyadmin/; secure; HttpOnly"],"X-Xss-Protection":["1; mode=block"],"Date":["Sat, 22 May 2021 05:08:41 GMT"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Type":["text/html; charset=utf-8"],"X-Powered-By":["PHP/7.4.16"],"X-Frame-Options":["DENY"],"X-Permitted-Cross-Domain-Policies":["none"],"Content-Encoding":["gzip"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Last-Modified":["Sat, 22 May 2021 05:08:41 +0000"],"X-Content-Type-Options":["nosniff"],"Expires":["Sat, 22 May 2021 05:08:41 +0000"],"Server":["Caddy"],"Vary":["Accept-Encoding"]},"status":200}

Backend log:

{"level":"debug","ts":"2021-05-22T13:08:40.167+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"],"Purpose":["prefetch"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.1.1.222"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-22T13:08:40.168+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"],"Purpose":["prefetch"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8","HTTP_HOST":"test.lan:443","HTTP_PURPOSE":"prefetch","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"13619","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T13:08:40.411+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Purpose":["prefetch"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"X-Frame-Options":["DENY"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Expires":["Sat, 22 May 2021 05:08:40 +0000"],"Last-Modified":["Sat, 22 May 2021 05:08:40 +0000"],"X-Ob_mode":["1"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Content-Type-Options":["nosniff"],"X-Permitted-Cross-Domain-Policies":["none"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Pragma":["no-cache"],"Content-Encoding":["gzip"],"Set-Cookie":["phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e; path=/phpmyadmin/; secure; HttpOnly"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Referrer-Policy":["no-referrer"],"X-Xss-Protection":["1; mode=block"],"X-Robots-Tag":["noindex, nofollow"],"Content-Type":["text/html; charset=utf-8"],"Vary":["Accept-Encoding"]},"status":200}
{"level":"debug","ts":"2021-05-22T13:08:41.306+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"X-Forwarded-Proto":["https"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-22T13:08:41.307+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Proto":["https"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"13619","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T13:08:41.543+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Proto":["https"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Referrer-Policy":["no-referrer"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Ob_mode":["1"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"],"Expires":["Sat, 22 May 2021 05:08:41 +0000"],"Content-Type":["text/html; charset=utf-8"],"Set-Cookie":["phpMyAdmin_https=h65e8g79vlpcums8lpsjq0h9b7; path=/phpmyadmin/; secure; HttpOnly"],"X-Xss-Protection":["1; mode=block"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Encoding":["gzip"],"X-Powered-By":["PHP/7.4.16"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Robots-Tag":["noindex, nofollow"],"Pragma":["no-cache"],"Last-Modified":["Sat, 22 May 2021 05:08:41 +0000"],"Vary":["Accept-Encoding"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Permitted-Cross-Domain-Policies":["none"]},"status":200}

Rob789 · May 22, 2021, 5:46pm

Accessing the backend host directly doesn’t work for me either but this is expected. Unless you install the root CA in your browser.

Iv’e setup my split-DNS to redirect to the frontend Caddy ie nextcloud.mydomain.com goes straight to 192.168.2.2

Just an extra warning; I lost a lot of time debugging a working config because my browsers had something in the cache that would prevent a successful connection. I also had very different (cache) behaviour between Chrome and Firefox. I think the best results for me was to clear the browser cache and then close all browser windows before reconnecting again.

I also noticed that Caddy can get into a condition where the certificate renewal doesn’t work correctly:

I empty the Caddy storage ie rm -rf /.local/share on both the frontend and backend
I restart Caddy frontend to generate a new root CA for the internal ACME server
I copy the new certificate to the backend
I start Caddy on the backend
New certificates are being issues but connecting to the services gives me a certificate error similar to

"x509: certificate signed by unknown authority"

but there was an additional message which I lost and although I could reproduce this 3 times, not anymore…

Restarting Caddy may solve this but I don’t have solid proof (yet). When I do I’ll report this in a separate topic.