mTLS under FreeBSD

It’s essentially the HOME environment variable of the user under which you ran Caddy.

Yeah, that’s harmless. Ignore it. In a later version, we’ll have an option to skip the install step which it tries at startup.

Did you also copy that newly found root cert to your backend Caddy instance too? It also needs that one to trust the frontend.

You might need to wipe the backend instance’s data storage to force it to fetch a new cert from the frontend with the right trust established maybe.

Yeah that looks pretty good. :thinking:

Maybe there’s another step you need to take to install the root cert, cause Caddy doesn’t use openssl - it uses the implementation of crypto from the Go standard library.

I feel like I read in some cursory Googling that FreeBSD uses Mozilla’s NSS tools as the trust store. Maybe there’s something else to be done there on the frontend.

Or, you could configure reverse_proxy’s http transport with tls_trusted_ca_certs instead of mucking with the trust store. Less nice if a solution though.

Yes, I did. :+1:

Can you elaborate, please? I’m not exactly sure what’s involved.

I’ve mucked around so much with Caddy and Smallstep in the same space, I might have messed something up. I think I’m going to start with a fresh Caddy instance and see if I end up in the same place. This will give me the opportunity to upgrade to Caddy 2.4.0. With a bit of luck, it may just be something I’ve inadvertently broken. If the issue is reproducible, I’ll then reflect on this advice.

Basically just rm -rf the contents of Caddy’s storage, then restart Caddy. But doing a fresh install avoids needing to do that :+1:

I rebuilt the frontend Caddy instance, but still end up with the ‘x509: certificate signed by unknown authority’ error…

root@caddy:~ # caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

root@caddy:~ # tail --lines=1 /var/log/caddy.log
{"level":"error","ts":"2021-05-20T04:46:11.333+0800","logger":"http.log.error.log4","msg":"x509: certificate signed by unknown authority","request":{"remote_addr":"108.162.221.21:60838","proto":"HTTP/1.1","method":"HEAD","host":"test.udance.com.au","uri":"/","headers":{"X-Forwarded-For":["192.0.91.177"],"Cf-Ray":["65202c54dc6dc81e-DFW"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Cf-Connecting-Ip":["192.0.91.177"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["US"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["0a27fa09070000c81e109a5000000001"],"Connection":["Keep-Alive"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"duration":0.006129711,"status":502,"err_id":"ghqnryw6i","err_trace":"reverseproxy.statusError (reverseproxy.go:852)"}

I deleted the backend Caddy storage, restarted Caddy and the log showed that a new cert was fetched from the frontend. Would this behaviour suggest that the root cert was correctly inserted in the system trust in the frontend Caddy instance?

{"level":"info","ts":"2021-05-20T04:24:26.146+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-20T04:24:26.147+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-20T04:24:26.149+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002bafc0"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T04:24:26.151+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621455866.2774534,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621455866.2815413,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-20T04:24:26.284+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002e48c0"}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T04:24:26.285+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T04:24:26.287+0800","msg":"serving initial configuration"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-20T04:24:26.288+0800","logger":"tls.obtain","msg":"acquiring lock","identifier":"test.lan"}
Successfully started Caddy (pid=39197) - Caddy is running in the background
{"level":"info","ts":"2021-05-20T04:24:26.309+0800","logger":"tls.obtain","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:26.620+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T04:24:27.120+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"tls-alpn-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-20T04:24:27.167+0800","logger":"tls","msg":"served key authentication certificate","server_name":"test.lan","challenge":"tls-alpn-01","remote":"10.1.1.4:26438","distributed":false}
{"level":"info","ts":"2021-05-20T04:24:27.666+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/m9Oisrd24B6ktWgqfg1lxbLamzzuEFY3"}
{"level":"info","ts":"2021-05-20T04:24:27.957+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/i9QC6UPnO0BdLB2U1QP1LSmhL0WO5OA1"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T04:24:27.958+0800","logger":"tls.obtain","msg":"releasing lock","identifier":"test.lan"}
{"level":"warn","ts":"2021-05-20T04:24:27.960+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}

If backend behaviour suggests that the cert was correctly installed in the frontend system trust, is it time to comment in the Smallstep repo?

ca_root_nss is one of the installed packages.

root@caddy:~ # pkg info
bash-5.1.4_1                   GNU Project's Bourne Again SHell
ca_root_nss-3.63               Root certificate bundle from the Mozilla Project
curl-7.76.0                    Command line tool and library for transferring data with URLs
cvsps-2.1_2                    Create patchset information from CVS
expat-2.2.10                   XML 1.0 parser written in C
gettext-runtime-0.21           GNU gettext runtime libraries and programs
git-2.31.1_1                   Distributed source code management tool
go-1.16.4,1                    Go programming language
indexinfo-0.3.1                Utility to regenerate the GNU info page index
libffi-3.3_1                   Foreign Function Interface
libnghttp2-1.43.0              HTTP/2.0 C Library
nano-5.5                       Nano's ANOther editor, an enhanced free Pico clone
p5-Authen-SASL-2.16_1          Perl5 module for SASL authentication
p5-CGI-4.51                    Handle Common Gateway Interface requests and responses
p5-Clone-0.45                  Recursively copy Perl datatypes
p5-Digest-HMAC-1.03_1          Perl5 interface to HMAC Message-Digest Algorithms
p5-Encode-Locale-1.05          Determine the locale encoding
p5-Error-0.17029               Error/exception handling in object-oriented programming style
p5-GSSAPI-0.28_1               Perl extension providing access to the GSSAPIv2 library
p5-HTML-Parser-3.75            Perl5 module for parsing HTML documents
p5-HTML-Tagset-3.20_1          Some useful data table in parsing HTML
p5-HTTP-Date-6.05              Conversion routines for the HTTP protocol date formats
p5-HTTP-Message-6.28           Representation of HTTP style messages
p5-IO-HTML-1.001_1             Open an HTML file with automatic charset detection
p5-IO-Socket-INET6-2.72_1      Perl module with object interface to AF_INET6 domain sockets
p5-IO-Socket-SSL-2.070         Perl5 interface to SSL sockets
p5-LWP-MediaTypes-6.04         Guess media type for a file or a URL
p5-Mozilla-CA-20200520         Perl extension for Mozilla CA cert bundle in PEM format
p5-Net-SSLeay-1.88             Perl5 interface to SSL
p5-Socket6-0.29                IPv6 related part of the C socket.h defines and structure manipulators
p5-TimeDate-2.33,1             Perl5 module containing a better/faster date parser for absolute dates
p5-URI-5.07                    Perl5 interface to Uniform Resource Identifier (URI) references
pcre2-10.36                    Perl Compatible Regular Expressions library, version 2
perl5-5.32.1_1                 Practical Extraction and Report Language
pkg-1.16.3                     Package manager
python37-3.7.10                Interpreted object-oriented programming language
readline-8.1.0                 Library for editing command lines as they are typed

Is it time to consider this, or exhaust other options first e.g. try to understand and resolve the observed frontend error?

When comparing the frontend and backend Caddy logs, it’s pretty clear that the frontend is not communicating with the backend when the error occurs.

No, that just means that you used the right certificate for acme_ca_root on the backend. The frontend’s system trust is not involved when actually issuing certificates to other Caddy instances. It’s involved when actually proxying requests to the other instances.

It’s time to comment there as soon as you have something useful to report to them in terms of how best to install root CA certs on FreeBSD.

Good to know. So you’ll need to figure out how to add your own certs to that bundle.

I mean, you can try it at any point. If it works, then you know that it’s an option.

Right, cause the TLS handshake fails before the backend decides to log anything about the request. If you enable the debug global option, you might see some logs on the backend though.

Well, 16 days later and we have liftoff! :cowboy_hat_face:…sort of :thinking:

The clues that led me to a ‘solution’…
I looked a little deeper into ca_root_nss:

root@caddy:~ # pkg info -l ca_root_nss
ca_root_nss-3.63:
        /etc/ssl/cert.pem
        /usr/local/etc/ssl/cert.pem.sample
        /usr/local/openssl/cert.pem.sample
        /usr/local/share/certs/ca-root-nss.crt
        /usr/local/share/licenses/ca_root_nss-3.63/LICENSE
        /usr/local/share/licenses/ca_root_nss-3.63/MPL20
        /usr/local/share/licenses/ca_root_nss-3.63/catalog.mk

…and then appended root.crt to ca-root-nss.crt:

root@caddy:~ # cat /.local/share/caddy/pki/authorities/local/root.crt >> /usr/local/share/certs/ca-root-nss.crt

I then restarted frontend Caddy and noticed a change in the log regarding the trust of the root cert:

{"level":"info","ts":"2021-05-20T13:13:14.114+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-20T13:13:14.114+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-20T13:13:14.117+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000315d50"}
{"level":"info","ts":"2021-05-20T13:13:14.118+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T13:13:14.118+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621487594.170676,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621487594.1781077,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-20T13:13:14.181+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-20T13:13:14.181+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003ce5b0"}
{"level":"info","ts":"2021-05-20T13:13:14.191+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:13:14.191+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["xenografix.com.au","acme.lan","www.readymcgetty.com.au","www.caffigoalkeeping.com.au","readymcgetty.com.au","www.udance.com.au","udance.com.au","www.xenografix.com.au","caffigoalkeeping.com","*.udance.com.au","www.caffigoalkeeping.com","caffigoalkeeping.com.au"]}
{"level":"info","ts":"2021-05-20T13:13:20.993+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"warn","ts":"2021-05-20T13:13:20.994+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:13:20.997+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-20T13:13:21.001+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:13:21.001+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=64612) - Caddy is running in the background

Accessing the test site… Nice! :smiley:


However, I don’t see anything in the frontend or backend Caddy logs unless I turn debug on. It would be nice to have some basic logging happening for mTLS.

The system trust solution I stumbled upon is semi-permanent at best. The clue was in this FreeBSD bug report Bug 160387 - security/ca_root_nss: Allow user to trust extra local certificates. The trust breaks if the ca_root_nss package is updated. To fix it, it’s necessary to append root.crt to ca-root-nss.crt again. For reasons described in the bug report, a permanent solution is unlikely to be forthcoming.

Question: How do I convince myself that the communication path between the backend and frontend Caddy services is actually encrypted?

:rocket::rocket::rocket::rocket::rocket::rocket::rocket::rocket:

The reverse_proxy logs should show that the request has TLS on it. If it does, then you’re good to go. Not much else to say, really. There’s nothing inherent about mTLS that can be logged, it’s just regular HTTPS proxying, plus non-public trust. That’s all.

Figures. That’s kinda annoying. There might be a way to hook into your package manager to run some script after each time a package is updated… maybe? I know nothing of FreeBSD’s package manager.

But you could report this on the smallstep repo. Should be helpful.

The euphoria was somewhat short-lived. I’m no longer able to access the test site. :sob:

In the few hours I’ve been away, this is what’s accumulated in the logs:

Backend Caddy log:

{"level":"info","ts":"2021-05-20T13:39:29.599+0800","logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:52184","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1684"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":"2021-05-20T13:39:29.600+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
{"level":"info","ts":"2021-05-20T13:39:29.601+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:39:29.601+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:39:29.603+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"warn","ts":"2021-05-20T13:39:29.604+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:39:29.604+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002ca700"}
{"level":"info","ts":"2021-05-20T13:39:30.662+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002c5570"}
{"level":"info","ts":"2021-05-20T13:39:30.663+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:39:30.663+0800","logger":"admin.api","msg":"load complete"}
{"level":"info","ts":"2021-05-20T13:39:31.208+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-20T20:39:29.606+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["test.lan"],"remaining":14017.393058953}
{"level":"info","ts":"2021-05-20T20:39:29.607+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["test.lan"],"remaining":14017.392423428}
{"level":"info","ts":"2021-05-20T20:39:29.608+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:29.640+0800","logger":"tls.renew","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:29.640+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"test.lan","remaining":14017.359107754}
{"level":"info","ts":"2021-05-20T20:39:29.642+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T20:39:29.642+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-20T20:39:29.998+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-20T20:39:30.055+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:47482","distributed":false}
{"level":"info","ts":"2021-05-20T20:39:30.404+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri"}
{"level":"info","ts":"2021-05-20T20:39:30.583+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/fPbBOhtqkDfWJOdMMvB2B6kKNVTeAvAQ"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls.renew","msg":"releasing lock","identifier":"test.lan"}
{"level":"info","ts":"2021-05-20T20:39:30.584+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["test.lan"]}
{"level":"warn","ts":"2021-05-20T20:39:30.586+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T20:39:30.586+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["test.lan"],"new_expiration":"2021-05-21T00:39:30.000Z"}

Frontend Caddy log - noticeable events.

{"level":"info","ts":"2021-05-20T13:39:55.408+0800","logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:27757","headers":{"Accept-Encoding":["gzip"],"Content-Length":["10520"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
{"level":"info","ts":"2021-05-20T13:39:55.412+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-20T13:39:55.412+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003060e0"}
{"level":"info","ts":"2021-05-20T13:39:55.413+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-20T13:39:55.413+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-20T13:40:01.692+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.caffigoalkeeping.com.au","udance.com.au","www.readymcgetty.com.au","www.xenografix.com.au","www.udance.com.au","*.udance.com.au","caffigoalkeeping.com.au","readymcgetty.com.au","acme.lan","www.caffigoalkeeping.com","caffigoalkeeping.com","xenografix.com.au"]}
{"level":"warn","ts":"2021-05-20T13:40:01.699+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T13:40:01.699+0800","logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000178460"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-20T13:40:02.263+0800","logger":"admin.api","msg":"load complete"}
{"level":"info","ts":"2021-05-20T13:40:02.344+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}

...

{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.cache.maintenance","msg":"certificate expires soon; queuing for renewal","identifiers":["acme.lan"],"remaining":14226.584475591}
{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.cache.maintenance","msg":"attempting certificate renewal","identifiers":["acme.lan"],"remaining":14226.584391609}
{"level":"info","ts":"2021-05-20T19:49:55.415+0800","logger":"tls.renew","msg":"acquiring lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.471+0800","logger":"tls.renew","msg":"lock acquired","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.471+0800","logger":"tls.renew","msg":"renewing certificate","identifier":"acme.lan","remaining":14226.528197201}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls.renew","msg":"certificate renewed successfully","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls.renew","msg":"releasing lock","identifier":"acme.lan"}
{"level":"info","ts":"2021-05-20T19:49:55.473+0800","logger":"tls","msg":"reloading managed certificate","identifiers":["acme.lan"]}
{"level":"warn","ts":"2021-05-20T19:49:55.474+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-20T19:49:55.474+0800","logger":"tls.cache","msg":"replaced certificate in cache","identifiers":["acme.lan"],"new_expiration":"2021-05-20T23:49:55.000Z"}

...

2021/05/20 20:39:29 {"newNonce":"https://acme.lan/acme/local/new-nonce","newAccount":"https://acme.lan/acme/local/new-account","newOrder":"https://acme.lan/acme/local/new-order","revokeCert":"https://acme.lan/acme/local/revoke-cert","keyChange":"https://acme.lan/acme/local/key-change"}
2021/05/20 20:39:29 {"status":"pending","expires":"2021-05-21T12:39:30Z","identifiers":[{"type":"dns","value":"test.lan"}],"notBefore":"2021-05-20T12:39:30Z","notAfter":"2021-05-21T00:39:30Z","authorizations":["https://acme.lan/acme/local/authz/niNSkBvVy4bwYMeKQvS48pfBWzgw4DRY"],"finalize":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri/finalize"}
2021/05/20 20:39:29 {"identifier":{"type":"dns","value":"test.lan"},"status":"pending","expires":"2021-05-21T12:39:30Z","challenges":[{"type":"http-01","status":"pending","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"},{"type":"tls-alpn-01","status":"pending","token":"7tjZva3KtSzzdnGtK6YPOzNoasbiVcZW","url":"https://acme.lan/acme/local/challenge/yVYPnacjhH6QmzUhZpjNPOfTGPEQWHwV"},{"type":"dns-01","status":"pending","token":"CQzbgaNLQdMzir80pqYnQtX74U0tRNod","url":"https://acme.lan/acme/local/challenge/mBUvK1duKwkNj1VeQ3ysh4POvgKfqceU"}],"wildcard":false}
2021/05/20 20:39:30 {"type":"http-01","status":"valid","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","validated":"2021-05-20T12:39:30Z","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"}
2021/05/20 20:39:30 {"identifier":{"type":"dns","value":"test.lan"},"status":"valid","expires":"2021-05-21T12:39:30Z","challenges":[{"type":"http-01","status":"valid","token":"8Lu0YNe71ATVmHZAtsplqDJKBFID68WY","validated":"2021-05-20T12:39:30Z","url":"https://acme.lan/acme/local/challenge/YwaVOeDZr18tTbkt0zGQSUliO7QgNyOh"},{"type":"tls-alpn-01","status":"pending","token":"7tjZva3KtSzzdnGtK6YPOzNoasbiVcZW","url":"https://acme.lan/acme/local/challenge/yVYPnacjhH6QmzUhZpjNPOfTGPEQWHwV"},{"type":"dns-01","status":"pending","token":"CQzbgaNLQdMzir80pqYnQtX74U0tRNod","url":"https://acme.lan/acme/local/challenge/mBUvK1duKwkNj1VeQ3ysh4POvgKfqceU"}],"wildcard":false}
2021/05/20 20:39:30 {"status":"valid","expires":"2021-05-21T12:39:30Z","identifiers":[{"type":"dns","value":"test.lan"}],"notBefore":"2021-05-20T12:39:30Z","notAfter":"2021-05-21T00:39:30Z","authorizations":["https://acme.lan/acme/local/authz/niNSkBvVy4bwYMeKQvS48pfBWzgw4DRY"],"finalize":"https://acme.lan/acme/local/order/8UoAMZNziYQ3LHoxHtZnMYhfvIE0ybri/finalize","certificate":"https://acme.lan/acme/local/certificate/fPbBOhtqkDfWJOdMMvB2B6kKNVTeAvAQ"}

...

There’s a pattern. In the frontend log, you can match up the first and third groups with what’s in the backend log. Interestingly, the format of the third group changes in the log, but then returns to normal in the messages that follow this group.I’m not sure if there’s any significance in this.

When I attempt to acces the site externally, I can a redirection from test.udance.com.au to test.lan.

What’s probably more telling is when I access test.udance.com.au from the local network (split DNS arrangement). There’s a privacy error and a message about an invalid CA.

:man_shrugging:

I’m not really gleaning anything useful from that. Make sure DNS is correct, make sure the root CA certificate is still in the NSS bundle (did it update since last night lol)?

The logs look pretty normal aside from that change in format, which I can’t explain.

It all checks out. The way I recovered was to clear the backend Caddy storage and restart Caddy so it would acquire a new certificate.

{"level":"info","ts":"2021-05-21T00:17:26.889+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-21T00:17:26.889+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-21T00:17:26.892+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000182770"}
{"level":"info","ts":"2021-05-21T00:17:26.893+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-21T00:17:26.894+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1621527447.0243363,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1621527447.0288177,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-21T00:17:27.032+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-21T00:17:27.033+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-21T00:17:27.033+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-21T00:17:27.033+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002eca10"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-21T00:17:27.035+0800","msg":"serving initial configuration"}
{"level":"info","ts":"2021-05-21T00:17:27.036+0800","logger":"tls.obtain","msg":"acquiring lock","identifier":"test.lan"}
Successfully started Caddy (pid=21920) - Caddy is running in the background
{"level":"info","ts":"2021-05-21T00:17:27.072+0800","logger":"tls.obtain","msg":"lock acquired","identifier":"test.lan"}
{"level":"info","ts":"2021-05-21T00:17:27.425+0800","logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-21T00:17:27.425+0800","logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test.lan"]}
{"level":"info","ts":"2021-05-21T00:17:27.681+0800","logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test.lan","challenge_type":"http-01","ca":"https://acme.lan/acme/local/directory"}
{"level":"info","ts":"2021-05-21T00:17:27.728+0800","logger":"tls.issuance.acme","msg":"served key authentication","identifier":"test.lan","challenge":"http-01","remote":"10.1.1.4:23813","distributed":false}
{"level":"info","ts":"2021-05-21T00:17:28.093+0800","logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.lan/acme/local/order/Yvy89ZiAowVe5FrW4b1vZUEm7SGMamuH"}
{"level":"info","ts":"2021-05-21T00:17:28.306+0800","logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme.lan/acme/local/certificate/5CiXEmiAu0u3oOf17Rdne0iK3ThKobb0"}
{"level":"info","ts":"2021-05-21T00:17:28.307+0800","logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"test.lan"}
{"level":"info","ts":"2021-05-21T00:17:28.307+0800","logger":"tls.obtain","msg":"releasing lock","identifier":"test.lan"}
{"level":"warn","ts":"2021-05-21T00:17:28.309+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}

I also had to clear browser caches as the browsers seemed to remember the redirection to test.lan. I’ll monitor this over the next few hours and see if the issue occurs again.

1 Like

The mTLS service I got working yesterday is extremely fragile. It collapsed a few minutes after I got it working last night and I haven’t been able to resurrect it since. So, a hard reset…

  1. I rebuilt the Caddy front end.
  2. I added the root.crt to the NSS bundle in the frontend and copied it to the backend as well.
  3. I cleared Caddy storage in the backend.
  4. I turned debug on in both the frontend and backend Caddyfiles.
  5. I restarted both Caddy frontend and backend services.
  6. I cleared my browser cache.

I then attempted to access the test site (test.udance.com.au) on the local network and immediately got directed to test.lan.

The local DNS resolves acme.lan → 10.1.1.4 and test.lan → 10.1.1.50. My local machine has an IP of 10.1.1.222.

This is what appeared in the backend Caddy log at the time I accessed the test site.

{"level":"debug","ts":"2021-05-21T11:55:22.637+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-For":["10.1.1.222"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-21T11:55:22.637+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Gpc":["1"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_SEC_GPC":"1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"25859","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-21T11:55:24.285+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","headers":{"Accept-Encoding":["deflate, gzip"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Content-Length":["0"],"Accept":["*/*"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621569324.2422480583190917968750","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"25859","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-21T11:55:24.399+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","headers":{"Accept-Encoding":["deflate, gzip"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Content-Length":["0"],"Accept":["*/*"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"]},"status":200}
{"level":"debug","ts":"2021-05-21T11:55:24.428+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:25859","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Gpc":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}
{"level":"debug","ts":"2021-05-21T11:55:24.494+0800","logger":"http.stdlib","msg":"http: TLS handshake error from 10.1.1.222:64614: remote error: tls: unknown certificate"}

…and in the frontend Caddy log…

{"level":"debug","ts":"2021-05-21T11:55:24.399+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:30648","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750","headers":{"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621569324.2422480583190917968750"],"X-Forwarded-Proto":["https"],"Content-Length":["0"],"X-Forwarded-Host":["test.udance.com.au"],"Accept-Encoding":["deflate, gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"],"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Fri, 21 May 2021 03:55:24 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-21T11:55:24.427+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:64602","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"Sec-Gpc":["1"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-For":["10.1.1.222"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko)Chrome/90.0.4430.212 Safari/537.36"],"X-Forwarded-Host":["test.udance.com.au"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"heimdall.udance.com.au"}},"headers":{"Date":["Fri, 21 May 2021 03:55:24 GMT"],"Content-Type":["text/html; charset=UTF-8"],"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"X-Redirect-By":["WordPress"],"Content-Length":["0"]},"status":301}

From my reckoning, things seem to collapse starting with the last entry in the frontend log (time T11:55:24.427). There’s a reference to heimdall.udance.com.au (IP 10.1.1.23)…I’m not sure why? From that time forward it all seems to fall apart in the backend log (the last two entries).

Unrelated, but probably worth mentioning…When I rebuilt the Caddy frontend this morning using xcaddy including the Cloudflare module, Caddy got upgraded:

root@caddy:~ # caddy version
v2.4.1 h1:kAJ0JB5Xk5gPdTH/27S5cyoMGqD5lBAe9yZ8zTjVJa0=

On the backend, I use a Caddy static binary. Running caddy upgrade does not upgrade the binary.

root@wordpress:~ # caddy upgrade
2021/05/21 05:21:02.466 INFO    this executable will be replaced        {"path": "/usr/local/bin/caddy"}
2021/05/21 05:21:02.467 INFO    requesting build        {"os": "freebsd", "arch": "amd64", "packages": []}
2021/05/21 05:21:03.484 INFO    build acquired; backing up current executable   {"current_path": "/usr/local/bin/caddy", "backup_path": "/usr/local/bin/caddy.tmp"}
2021/05/21 05:21:03.485 INFO    downloading binary      {"source": "https://caddyserver.com/api/download?arch=amd64&os=freebsd", "destination": "/usr/local/bin/caddy"}
2021/05/21 05:21:16.055 INFO    download successful; displaying new binary details      {"location": "/usr/local/bin/caddy"}

Module versions:

admin.api.load v2.4.0
admin.api.metrics v2.4.0
admin.api.reverse_proxy v2.4.0
caddy.adapters.caddyfile v2.4.0
caddy.config_loaders.http v2.4.0
caddy.listeners.tls v2.4.0
caddy.logging.encoders.console v2.4.0
caddy.logging.encoders.filter v2.4.0
caddy.logging.encoders.filter.delete v2.4.0
caddy.logging.encoders.filter.ip_mask v2.4.0
caddy.logging.encoders.filter.replace v2.4.0
caddy.logging.encoders.json v2.4.0
caddy.logging.encoders.single_field v2.4.0
caddy.logging.writers.discard v2.4.0
caddy.logging.writers.file v2.4.0
caddy.logging.writers.net v2.4.0
caddy.logging.writers.stderr v2.4.0
caddy.logging.writers.stdout v2.4.0
caddy.storage.file_system v2.4.0
http v2.4.0
http.authentication.hashes.bcrypt v2.4.0
http.authentication.hashes.scrypt v2.4.0
http.authentication.providers.http_basic v2.4.0
http.encoders.gzip v2.4.0
http.encoders.zstd v2.4.0
http.handlers.acme_server v2.4.0
http.handlers.authentication v2.4.0
http.handlers.encode v2.4.0
http.handlers.error v2.4.0
http.handlers.file_server v2.4.0
http.handlers.headers v2.4.0
http.handlers.map v2.4.0
http.handlers.metrics v2.4.0
http.handlers.push v2.4.0
http.handlers.request_body v2.4.0
http.handlers.reverse_proxy v2.4.0
http.handlers.rewrite v2.4.0
http.handlers.static_response v2.4.0
http.handlers.subroute v2.4.0
http.handlers.templates v2.4.0
http.handlers.vars v2.4.0
http.matchers.expression v2.4.0
http.matchers.file v2.4.0
http.matchers.header v2.4.0
http.matchers.header_regexp v2.4.0
http.matchers.host v2.4.0
http.matchers.method v2.4.0
http.matchers.not v2.4.0
http.matchers.path v2.4.0
http.matchers.path_regexp v2.4.0
http.matchers.protocol v2.4.0
http.matchers.query v2.4.0
http.matchers.remote_ip v2.4.0
http.matchers.vars v2.4.0
http.matchers.vars_regexp v2.4.0
http.precompressed.br v2.4.0
http.precompressed.gzip v2.4.0
http.precompressed.zstd v2.4.0
http.reverse_proxy.selection_policies.cookie v2.4.0
http.reverse_proxy.selection_policies.first v2.4.0
http.reverse_proxy.selection_policies.header v2.4.0
http.reverse_proxy.selection_policies.ip_hash v2.4.0
http.reverse_proxy.selection_policies.least_conn v2.4.0
http.reverse_proxy.selection_policies.random v2.4.0
http.reverse_proxy.selection_policies.random_choose v2.4.0
http.reverse_proxy.selection_policies.round_robin v2.4.0
http.reverse_proxy.selection_policies.uri_hash v2.4.0
http.reverse_proxy.transport.fastcgi v2.4.0
http.reverse_proxy.transport.http v2.4.0
pki v2.4.0
tls v2.4.0
tls.certificates.automate v2.4.0
tls.certificates.load_files v2.4.0
tls.certificates.load_folders v2.4.0
tls.certificates.load_pem v2.4.0
tls.certificates.load_storage v2.4.0
tls.handshake_match.remote_ip v2.4.0
tls.handshake_match.sni v2.4.0
tls.issuance.acme v2.4.0
tls.issuance.internal v2.4.0
tls.issuance.zerossl v2.4.0
tls.stek.distributed v2.4.0
tls.stek.standard v2.4.0

  Standard modules: 83

  Non-standard modules: 0

  Unknown modules: 0

Version:
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

2021/05/21 05:21:16.218 INFO    upgrade successful; please restart any running Caddy instances  {"executable": "/usr/local/bin/caddy"}
root@wordpress:~ # service caddy restart
Stopping caddy.
Waiting for PIDS: 32125.
root@wordpress:~ # caddy version
v2.4.0 h1:yHnnbawH2G3ZBP2mAJF4XBLnJanqhULLP/wu01Qi9Io=

The next thing to try was to reset the local DNS resolver (DNSMasq on DD-WRT) just in case there was something weird happening in that space.

Attempting to access the test site test.udance.com.au externally:

This is what appeared in the logs:

Frontend:

{"level":"debug","ts":"2021-05-22T11:56:04.728+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:21924","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","headers":{"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-For":["10.1.1.50"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125"],"X-Forwarded-Proto":["https"],"Content-Length":["0"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"],"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Sat, 22 May 2021 03:56:04 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-22T11:56:04.757+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"172.70.49.149:43108","proto":"HTTP/1.1","method":"GET","host":"test.lan:443","uri":"/","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Cf-Ray":["65331cc398700a76-KIX"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua-Mobile":["?1"],"Save-Data":["on"],"Cdn-Loop":["cloudflare"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"Cf-Ipcountry":["AU"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Accept-Encoding":["gzip"],"Cf-Connecting-Ip":["49.196.150.225"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Date":["Sat, 22 May 2021 03:56:04 GMT"],"Content-Type":["text/html; charset=UTF-8"],"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"X-Redirect-By":["WordPress"],"Content-Length":["0"]},"status":301}

Backend:

{"level":"debug","ts":"2021-05-22T11:56:02.905+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-Site":["none"],"Cf-Ray":["65331cc398700a76-KIX"],"Sec-Fetch-Mode":["navigate"],"Save-Data":["on"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"X-Forwarded-Proto":["https"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Accept-Encoding":["gzip"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.150.225"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-22T11:56:02.905+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-Site":["none"],"Cf-Ray":["65331cc398700a76-KIX"],"Sec-Fetch-Mode":["navigate"],"Save-Data":["on"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.150.225"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"49.196.150.225","HTTP_CF_IPCOUNTRY":"AU","HTTP_CF_RAY":"65331cc398700a76-KIX","HTTP_CF_REQUEST_ID":"0a33d04e4300000a768e190000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_COOKIE":"tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22","HTTP_HOST":"test.lan:443","HTTP_SAVE_DATA":"on","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?1","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36","HTTP_X_FORWARDED_FOR":"49.196.150.225, 172.70.49.149, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33756","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T11:56:04.612+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","headers":{"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Content-Length":["0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621655764.5685970783233642578125","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33756","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T11:56:04.726+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125","headers":{"Content-Length":["0"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621655764.5685970783233642578125"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"X-Powered-By":["PHP/7.4.16"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-22T11:56:04.756+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33756","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Dest":["document"],"Cookie":["tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-For":["49.196.150.225, 172.70.49.149, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.150.225"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-Site":["none"],"Cf-Ray":["65331cc398700a76-KIX"],"Sec-Fetch-Mode":["navigate"],"Save-Data":["on"],"Cf-Request-Id":["0a33d04e4300000a768e190000000001"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}

Repeating the exercise, but accessing the site internally (split-DNS):

This is what appears in the logs:

Frontend:

{"level":"debug","ts":"2021-05-22T12:10:07.719+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:37639","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","headers":{"Content-Length":["0"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept-Encoding":["deflate, gzip"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Sat, 22 May 2021 04:10:07 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"]},"status":200}
{"level":"debug","ts":"2021-05-22T12:10:07.747+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:59220","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Sec-Fetch-Dest":["document"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?0"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["test.udance.com.au"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["10.1.1.222"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Redirect-By":["WordPress"],"Content-Length":["0"],"Date":["Sat, 22 May 2021 04:10:07 GMT"],"Content-Type":["text/html; charset=UTF-8"],"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"]},"status":301}

Backend:

{"level":"debug","ts":"2021-05-22T12:10:05.992+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"X-Forwarded-For":["10.1.1.222"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-22T12:10:05.992+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33137","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T12:10:07.605+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Content-Length":["0"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept":["*/*"],"Accept-Encoding":["deflate, gzip"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621656607.5627350807189941406250","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"33137","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T12:10:07.719+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250","headers":{"Content-Length":["0"],"X-Forwarded-Proto":["https"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Accept":["*/*"],"X-Forwarded-Host":["test.udance.com.au"],"Accept-Encoding":["deflate, gzip"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621656607.5627350807189941406250"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"X-Powered-By":["PHP/7.4.16"]},"status":200}
{"level":"debug","ts":"2021-05-22T12:10:07.747+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:33137","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"Upgrade-Insecure-Requests":["1"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}
{"level":"debug","ts":"2021-05-22T12:10:07.792+0800","logger":"http.stdlib","msg":"http: TLS handshake error from 10.1.1.222:59223: remote error: tls: unknown certificate"}

EDIT: I believe why mTLS was so flaky when I first got it working was due to WordPress frontend cachiing on the test site. I’ve turned that off, but I’ve left WP backend object caching (Redis) on. Since then, I’m consistently getting the behaviour described in this post.

Breaking news! I can access other sub-paths of the test site without issue. It’s the subdomain level that has an issue (previous post).

Externally:

Frontend log:

{"level":"debug","ts":"2021-05-22T12:49:27.298+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"162.158.119.150:31132","proto":"HTTP/1.1","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Accept-Encoding":["gzip"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua-Mobile":["?1"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Save-Data":["on"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cf-Ray":["65336afc8d281d87-NRT"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Mode":["navigate"],"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"],"Cf-Ipcountry":["AU"],"Cf-Connecting-Ip":["49.196.150.225"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Permitted-Cross-Domain-Policies":["none"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Set-Cookie":["phpMyAdmin_https=ret9asq4j2diuif6utq6udgtcr; path=/phpmyadmin/; secure; HttpOnly"],"Referrer-Policy":["no-referrer"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["DENY"],"X-Robots-Tag":["noindex, nofollow"],"Date":["Sat, 22 May 2021 04:49:27 GMT"],"Content-Encoding":["gzip"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Last-Modified":["Sat, 22 May 2021 04:49:27 +0000"],"Server":["Caddy"],"X-Powered-By":["PHP/7.4.16"],"X-Xss-Protection":["1; mode=block"],"Content-Type":["text/html; charset=utf-8"],"Expires":["Sat, 22 May 2021 04:49:27 +0000"],"X-Ob_mode":["1"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Pragma":["no-cache"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"]},"status":200}

Backend log:

{"level":"debug","ts":"2021-05-22T12:49:27.050+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:27626","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["65336afc8d281d87-NRT"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?1"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Save-Data":["on"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"Accept-Encoding":["gzip"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Cf-Connecting-Ip":["49.196.150.225"],"Sec-Fetch-Mode":["navigate"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-22T12:49:27.051+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:27626","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["65336afc8d281d87-NRT"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Save-Data":["on"],"Cf-Connecting-Ip":["49.196.150.225"],"Sec-Fetch-Mode":["navigate"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"49.196.150.225","HTTP_CF_IPCOUNTRY":"AU","HTTP_CF_RAY":"65336afc8d281d87-NRT","HTTP_CF_REQUEST_ID":"0a340131d500001d87e5180000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F","HTTP_HOST":"test.lan:443","HTTP_SAVE_DATA":"on","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?1","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36","HTTP_X_FORWARDED_FOR":"49.196.150.225, 162.158.119.150, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"27626","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T12:49:27.300+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:27626","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Cf-Request-Id":["0a340131d500001d87e5180000000001"],"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["49.196.150.225, 162.158.119.150, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cdn-Loop":["cloudflare"],"Sec-Ch-Ua-Mobile":["?1"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["65336afc8d281d87-NRT"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.210 Mobile Safari/537.36"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=r539aufliecuvs7o9v067omu40; tk_or=%22%22; tk_r3d=%22%22; tk_lr=%22%22; wordpress_test_cookie=WP%20Cookie%20check; jetpack_sso_redirect_to=https%3A%2F%2Ftest.lan%3A443%2Fwp-admin%2F"],"Accept-Encoding":["gzip"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Save-Data":["on"],"Cf-Connecting-Ip":["49.196.150.225"],"Sec-Fetch-Mode":["navigate"],"Cf-Ipcountry":["AU"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Google Chrome\";v=\"90\""],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Expires":["Sat, 22 May 2021 04:49:27 +0000"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Referrer-Policy":["no-referrer"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Vary":["Accept-Encoding"],"X-Powered-By":["PHP/7.4.16"],"Set-Cookie":["phpMyAdmin_https=ret9asq4j2diuif6utq6udgtcr; path=/phpmyadmin/; secure; HttpOnly"],"X-Robots-Tag":["noindex, nofollow"],"Content-Type":["text/html; charset=utf-8"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Xss-Protection":["1; mode=block"],"X-Content-Type-Options":["nosniff"],"Pragma":["no-cache"],"Last-Modified":["Sat, 22 May 2021 04:49:27 +0000"],"Content-Encoding":["gzip"],"X-Ob_mode":["1"],"X-Frame-Options":["DENY"]},"status":200}

Internally:

Frontend log:

{"level":"debug","ts":"2021-05-22T13:08:40.408+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:60356","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["10.1.1.222"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Ch-Ua-Mobile":["?0"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Purpose":["prefetch"],"X-Forwarded-Host":["test.udance.com.au"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Encoding":["gzip"],"Expires":["Sat, 22 May 2021 05:08:40 +0000"],"X-Frame-Options":["DENY"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Robots-Tag":["noindex, nofollow"],"X-Xss-Protection":["1; mode=block"],"Last-Modified":["Sat, 22 May 2021 05:08:40 +0000"],"Pragma":["no-cache"],"Server":["Caddy"],"Vary":["Accept-Encoding"],"X-Content-Type-Options":["nosniff"],"X-Powered-By":["PHP/7.4.16"],"Date":["Sat, 22 May 2021 05:08:40 GMT"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Content-Type":["text/html; charset=utf-8"],"Referrer-Policy":["no-referrer"],"Set-Cookie":["phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e; path=/phpmyadmin/; secure; HttpOnly"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Ob_mode":["1"]},"status":200}
{"level":"debug","ts":"2021-05-22T13:08:41.540+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:60356","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"X-Forwarded-For":["10.1.1.222"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept-Language":["en-US,en;q=0.9"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Ob_mode":["1"],"X-Robots-Tag":["noindex, nofollow"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Pragma":["no-cache"],"Referrer-Policy":["no-referrer"],"Set-Cookie":["phpMyAdmin_https=h65e8g79vlpcums8lpsjq0h9b7; path=/phpmyadmin/; secure; HttpOnly"],"X-Xss-Protection":["1; mode=block"],"Date":["Sat, 22 May 2021 05:08:41 GMT"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Type":["text/html; charset=utf-8"],"X-Powered-By":["PHP/7.4.16"],"X-Frame-Options":["DENY"],"X-Permitted-Cross-Domain-Policies":["none"],"Content-Encoding":["gzip"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Last-Modified":["Sat, 22 May 2021 05:08:41 +0000"],"X-Content-Type-Options":["nosniff"],"Expires":["Sat, 22 May 2021 05:08:41 +0000"],"Server":["Caddy"],"Vary":["Accept-Encoding"]},"status":200}

Backend log:

{"level":"debug","ts":"2021-05-22T13:08:40.167+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"],"Purpose":["prefetch"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.1.1.222"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-22T13:08:40.168+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"],"Purpose":["prefetch"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8","HTTP_HOST":"test.lan:443","HTTP_PURPOSE":"prefetch","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"13619","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T13:08:40.411+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Purpose":["prefetch"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-Proto":["https"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=tu4a1331mcjpadv3lk87ijmta8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"X-Frame-Options":["DENY"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Expires":["Sat, 22 May 2021 05:08:40 +0000"],"Last-Modified":["Sat, 22 May 2021 05:08:40 +0000"],"X-Ob_mode":["1"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Content-Type-Options":["nosniff"],"X-Permitted-Cross-Domain-Policies":["none"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Pragma":["no-cache"],"Content-Encoding":["gzip"],"Set-Cookie":["phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e; path=/phpmyadmin/; secure; HttpOnly"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Referrer-Policy":["no-referrer"],"X-Xss-Protection":["1; mode=block"],"X-Robots-Tag":["noindex, nofollow"],"Content-Type":["text/html; charset=utf-8"],"Vary":["Accept-Encoding"]},"status":200}
{"level":"debug","ts":"2021-05-22T13:08:41.306+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"X-Forwarded-Proto":["https"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-22T13:08:41.307+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Proto":["https"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"13619","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.0","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-22T13:08:41.543+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:13619","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Proto":["https"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=b9surje51u90l3f0phqc2jmi4e"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.62"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Referrer-Policy":["no-referrer"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Ob_mode":["1"],"X-Frame-Options":["DENY"],"X-Content-Type-Options":["nosniff"],"Expires":["Sat, 22 May 2021 05:08:41 +0000"],"Content-Type":["text/html; charset=utf-8"],"Set-Cookie":["phpMyAdmin_https=h65e8g79vlpcums8lpsjq0h9b7; path=/phpmyadmin/; secure; HttpOnly"],"X-Xss-Protection":["1; mode=block"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Encoding":["gzip"],"X-Powered-By":["PHP/7.4.16"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Robots-Tag":["noindex, nofollow"],"Pragma":["no-cache"],"Last-Modified":["Sat, 22 May 2021 05:08:41 +0000"],"Vary":["Accept-Encoding"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Permitted-Cross-Domain-Policies":["none"]},"status":200}

Accessing the backend host directly doesn’t work for me either but this is expected. Unless you install the root CA in your browser.

Iv’e setup my split-DNS to redirect to the frontend Caddy ie nextcloud.mydomain.com goes straight to 192.168.2.2

Just an extra warning; I lost a lot of time debugging a working config because my browsers had something in the cache that would prevent a successful connection. I also had very different (cache) behaviour between Chrome and Firefox. I think the best results for me was to clear the browser cache and then close all browser windows before reconnecting again.

I also noticed that Caddy can get into a condition where the certificate renewal doesn’t work correctly:

I empty the Caddy storage ie rm -rf /.local/share on both the frontend and backend
I restart Caddy frontend to generate a new root CA for the internal ACME server
I copy the new certificate to the backend
I start Caddy on the backend
New certificates are being issues but connecting to the services gives me a certificate error similar to

"x509: certificate signed by unknown authority"

but there was an additional message which I lost and although I could reproduce this 3 times, not anymore…

Restarting Caddy may solve this but I don’t have solid proof (yet). When I do I’ll report this in a separate topic.

I do understand that, but that’s not what’s happening here. Strangely, when I access test.udance.com.au internally in my setup, I get a redirection to test.lan. Accessing sub-paths of test.udance.com.au are fine though.

Earlier, I was also getting a redirection to test.lan if I accessed test.udance.com.au externally. That’s since stopped and I now get a 502 error. Possibly, the difference is how quickly the CDN updates its caches.

Agree. I found that out the hard way too.

I’ve updated how Caddy is integrated with FreeBSD by making it much more compliant with the FreeBSD rc.d framework. The approach now also honours XDG_CONFIG_HOME/XDG_DATA_HOME described in Caddy documentation under File locations. Caddy now creates subdirectories for each of those under a /var/db/caddy root.

This is evidenced by the following sample lines in the process logs:

Frontend

{"level":"debug","ts":"2021-05-24T17:13:19.558+0800","logger":"tls","msg":"loading managed certificate","domain":"readymcgetty.com.au","expiration":"2021-08-22T02:29:03.000Z","issuer_key":"acme-v02.api.letsencrypt.org-directory","storage":"FileStorage:/var/db/caddy/data/caddy"}
{"level":"warn","ts":"2021-05-24T17:13:19.558+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-24T17:13:19.577+0800","logger":"pki.ca.local","msg":"root certificate is already trusted by system","path":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-24T17:13:19.577+0800","msg":"autosaved config (load with --resume flag)","file":"/var/db/caddy/config/caddy/autosave.json"}

Backend

{"level":"info","ts":"2021-05-24T17:18:36.565+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/db/caddy/data/caddy"}
{"level":"debug","ts":"2021-05-24T17:18:36.565+0800","logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
{"level":"info","ts":"2021-05-24T17:18:36.565+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["test.lan"]}
{"level":"info","ts":"2021-05-24T17:18:36.566+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"warn","ts":"2021-05-24T17:18:36.588+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [test.lan]: no OCSP server specified in certificate"}
{"level":"info","ts":"2021-05-24T17:18:36.588+0800","msg":"autosaved config (load with --resume flag)","file":"/var/db/caddy/config/caddy/autosave.json"}

I then repeated the tests in posts #59 and #60 to see if anything had changed. As evidenced below, the issues are still reproducible i.e. accessing the subdomain test.udance.com.au is a problem, but accessing its sub-paths is fine.

SUBDOMAIN TESTS

EXTERNAL

Accessing test.udance.com.au redirects to test.lan.

Frontend

{"level":"debug","ts":"2021-05-24T16:08:07.508+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:34106","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500","headers":{"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500"],"X-Forwarded-Proto":["https"],"Content-Length":["0"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Accept-Encoding":["deflate, gzip"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Server":["Caddy"],"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Mon, 24 May 2021 08:08:07 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-24T16:08:07.537+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"162.158.5.217:35172","proto":"HTTP/1.1","method":"GET","host":"test.lan:443","uri":"/","headers":{"Cdn-Loop":["cloudflare"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["654508b93cafaec7-KIX"],"Cf-Ipcountry":["AU"],"Cf-Request-Id":["0a3f03c7c00000aec792aac000000001"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-For":["49.196.36.201, 162.158.5.217"],"Sec-Fetch-Site":["none"],"Cookie":["tk_or=%22%22; tk_lr=%22%22"],"Accept-Encoding":["gzip"],"Cf-Connecting-Ip":["49.196.36.201"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Content-Length":["0"],"Date":["Mon, 24 May 2021 08:08:07 GMT"],"Content-Type":["text/html; charset=UTF-8"],"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"X-Redirect-By":["WordPress"]},"status":301}

Backend

{"level":"debug","ts":"2021-05-24T16:08:05.704+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:41591","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"Accept-Encoding":["gzip"],"Cf-Request-Id":["0a3f03c7c00000aec792aac000000001"],"Cookie":["tk_or=%22%22; tk_lr=%22%22"],"X-Forwarded-For":["49.196.36.201, 162.158.5.217"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"Cf-Ipcountry":["AU"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ray":["654508b93cafaec7-KIX"],"Sec-Fetch-Site":["none"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.36.201"],"Cdn-Loop":["cloudflare"],"Cf-Visitor":["{\"scheme\":\"https\"}"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-24T16:08:05.704+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:41591","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Cdn-Loop":["cloudflare"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.36.201"],"Cf-Request-Id":["0a3f03c7c00000aec792aac000000001"],"Accept-Encoding":["gzip"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"Cf-Ipcountry":["AU"],"Cookie":["tk_or=%22%22; tk_lr=%22%22"],"X-Forwarded-For":["49.196.36.201, 162.158.5.217, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Cf-Ray":["654508b93cafaec7-KIX"],"Sec-Fetch-Site":["none"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","HTTP_ACCEPT_ENCODING":"gzip","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"49.196.36.201","HTTP_CF_IPCOUNTRY":"AU","HTTP_CF_RAY":"654508b93cafaec7-KIX","HTTP_CF_REQUEST_ID":"0a3f03c7c00000aec792aac000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_COOKIE":"tk_or=%22%22; tk_lr=%22%22","HTTP_HOST":"test.lan:443","HTTP_SEC_FETCH_SITE":"none","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155","HTTP_X_FORWARDED_FOR":"49.196.36.201, 162.158.5.217, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"41591","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:08:07.393+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:41591","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500","headers":{"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Content-Length":["0"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["test.udance.com.au"],"Content-Type":["application/x-www-form-urlencoded"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621843687.3493719100952148437500","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"41591","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:08:07.509+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:41591","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Content-Type":["application/x-www-form-urlencoded"],"Accept":["*/*"],"X-Forwarded-Proto":["https"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Content-Length":["0"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621843687.3493719100952148437500"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"]},"status":200}
{"level":"debug","ts":"2021-05-24T16:08:07.538+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:41591","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"Cf-Ipcountry":["AU"],"Cookie":["tk_or=%22%22; tk_lr=%22%22"],"X-Forwarded-For":["49.196.36.201, 162.158.5.217, 10.1.1.4"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Cf-Ray":["654508b93cafaec7-KIX"],"Sec-Fetch-Site":["none"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cdn-Loop":["cloudflare"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Upgrade-Insecure-Requests":["1"],"Cf-Connecting-Ip":["49.196.36.201"],"Cf-Request-Id":["0a3f03c7c00000aec792aac000000001"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}

INTERNAL

Accessing test.udance.com.au redirects to test.lan .

mtls7

Frontend

{"level":"debug","ts":"2021-05-24T16:25:04.490+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.50:35992","proto":"HTTP/1.1","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750","headers":{"Content-Length":["0"],"X-Forwarded-For":["10.1.1.50"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750"],"X-Forwarded-Proto":["https"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Content-Length":["0"],"Date":["Mon, 24 May 2021 08:25:04 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"]},"status":200}
{"level":"debug","ts":"2021-05-24T16:25:04.519+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:49694","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"X-Forwarded-For":["10.1.1.222"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-User":["?1"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Location":["https://test.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"X-Redirect-By":["WordPress"],"Content-Length":["0"],"Date":["Mon, 24 May 2021 08:25:04 GMT"],"Content-Type":["text/html; charset=UTF-8"]},"status":301}
root@caddy:~ #

Backend

{"level":"debug","ts":"2021-05-24T16:25:02.700+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["10.1.1.222"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"],"Sec-Fetch-Site":["none"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-05-24T16:25:02.700+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"],"Sec-Fetch-Site":["none"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"],"X-Forwarded-Host":["test.udance.com.au"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"46355","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:25:04.376+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750","headers":{"Accept":["*/*"],"Content-Type":["application/x-www-form-urlencoded"],"Accept-Encoding":["deflate, gzip"],"X-Forwarded-Proto":["https"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750"],"Content-Length":["0"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621844704.3333621025085449218750","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"46355","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:25:04.490+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750","headers":{"X-Forwarded-Host":["test.udance.com.au"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621844704.3333621025085449218750"],"Content-Length":["0"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Content-Type":["application/x-www-form-urlencoded"],"Accept-Encoding":["deflate, gzip"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"X-Powered-By":["PHP/7.4.16"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"]},"status":200}
{"level":"debug","ts":"2021-05-24T16:25:04.519+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/index.php","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"],"Sec-Fetch-Site":["none"],"Sec-Ch-Ua-Mobile":["?0"],"Accept-Language":["en-US,en;q=0.9"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/html; charset=UTF-8"],"X-Redirect-By":["WordPress"],"Location":["https://test.lan/"]},"status":301}
{"level":"debug","ts":"2021-05-24T16:25:04.561+0800","logger":"http.stdlib","msg":"http: TLS handshake error from 10.1.1.222:49698: remote error: tls: unknown certificate"}

Continued in next post…

SUB-PATH TESTS

EXTERNAL

Frontend

{"level":"debug","ts":"2021-05-24T16:40:17.651+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"162.158.118.125:38708","proto":"HTTP/1.1","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["49.196.36.201, 162.158.118.125"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Cf-Request-Id":["0a3f21429700001d83410e3000000001"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Cdn-Loop":["cloudflare"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Cf-Ray":["654537e4284e1d83-NRT"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"Sec-Fetch-Mode":["navigate"],"Cf-Ipcountry":["AU"],"Cf-Connecting-Ip":["49.196.36.201"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=cpb9c2o1kbbs7s1skopdeh8u0i; tk_or=%22%22; tk_lr=%22%22"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Ob_mode":["1"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Date":["Mon, 24 May 2021 08:40:17 GMT"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Referrer-Policy":["no-referrer"],"Server":["Caddy"],"Set-Cookie":["phpMyAdmin_https=5sh31f5b1gpecd8pbjg3fgj6ml; path=/phpmyadmin/; secure; HttpOnly"],"Vary":["Accept-Encoding"],"X-Robots-Tag":["noindex, nofollow"],"X-Xss-Protection":["1; mode=block"],"Content-Type":["text/html; charset=utf-8"],"Last-Modified":["Mon, 24 May 2021 08:40:17 +0000"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Frame-Options":["DENY"],"X-Powered-By":["PHP/7.4.16"],"X-Permitted-Cross-Domain-Policies":["none"],"Content-Encoding":["gzip"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Expires":["Mon, 24 May 2021 08:40:17 +0000"],"Pragma":["no-cache"],"X-Content-Type-Options":["nosniff"]},"status":200}
{"level":"debug","ts":"2021-05-24T16:40:18.823+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"162.158.118.125:44186","proto":"HTTP/1.1","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/js/messages.php?l=en&v=5.1.0","headers":{"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Cf-Connecting-Ip":["49.196.36.201"],"Cf-Request-Id":["0a3f21476900001d833ba68000000001"],"Cf-Ipcountry":["AU"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["same-origin"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Cdn-Loop":["cloudflare"],"X-Forwarded-For":["49.196.36.201, 162.158.118.125"],"Cf-Ray":["654537ebdde41d83-NRT"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=5sh31f5b1gpecd8pbjg3fgj6ml; tk_or=%22%22; tk_lr=%22%22"],"Sec-Fetch-Mode":["no-cors"],"Accept":["*/*"],"Accept-Encoding":["gzip"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"Date":["Mon, 24 May 2021 08:40:18 GMT"],"Content-Encoding":["gzip"],"Content-Type":["text/javascript; charset=UTF-8"],"Expires":["Mon, 24 May 2021 09:40:18 GMT"],"Server":["Caddy"],"Vary":["Accept-Encoding"],"X-Ob_mode":["1"],"X-Powered-By":["PHP/7.4.16"]},"status":200}

Backend

{"level":"debug","ts":"2021-05-24T16:40:17.402+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["49.196.36.201, 162.158.118.125"],"Upgrade-Insecure-Requests":["1"],"Cf-Request-Id":["0a3f21429700001d83410e3000000001"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=cpb9c2o1kbbs7s1skopdeh8u0i; tk_or=%22%22; tk_lr=%22%22"],"Cf-Ray":["654537e4284e1d83-NRT"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Cf-Connecting-Ip":["49.196.36.201"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Sec-Fetch-User":["?1"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["none"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-24T16:40:17.403+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Cf-Ray":["654537e4284e1d83-NRT"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Cf-Connecting-Ip":["49.196.36.201"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Sec-Fetch-User":["?1"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["49.196.36.201, 162.158.118.125, 10.1.1.4"],"Upgrade-Insecure-Requests":["1"],"Cf-Request-Id":["0a3f21429700001d83410e3000000001"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=cpb9c2o1kbbs7s1skopdeh8u0i; tk_or=%22%22; tk_lr=%22%22"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3","HTTP_ACCEPT_ENCODING":"gzip","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"49.196.36.201","HTTP_CF_IPCOUNTRY":"AU","HTTP_CF_RAY":"654537e4284e1d83-NRT","HTTP_CF_REQUEST_ID":"0a3f21429700001d83410e3000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=cpb9c2o1kbbs7s1skopdeh8u0i; tk_or=%22%22; tk_lr=%22%22","HTTP_HOST":"test.lan:443","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155","HTTP_X_FORWARDED_FOR":"49.196.36.201, 162.158.118.125, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"46355","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:40:17.652+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"X-Forwarded-Proto":["https"],"Sec-Fetch-Site":["none"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=cpb9c2o1kbbs7s1skopdeh8u0i; tk_or=%22%22; tk_lr=%22%22"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"Accept-Encoding":["gzip"],"X-Forwarded-For":["49.196.36.201, 162.158.118.125, 10.1.1.4"],"Upgrade-Insecure-Requests":["1"],"Cf-Request-Id":["0a3f21429700001d83410e3000000001"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Sec-Fetch-User":["?1"],"Cf-Ray":["654537e4284e1d83-NRT"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Cf-Connecting-Ip":["49.196.36.201"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Cdn-Loop":["cloudflare"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Last-Modified":["Mon, 24 May 2021 08:40:17 +0000"],"Pragma":["no-cache"],"Set-Cookie":["phpMyAdmin_https=5sh31f5b1gpecd8pbjg3fgj6ml; path=/phpmyadmin/; secure; HttpOnly"],"X-Frame-Options":["DENY"],"Referrer-Policy":["no-referrer"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Content-Type-Options":["nosniff"],"X-Permitted-Cross-Domain-Policies":["none"],"Expires":["Mon, 24 May 2021 08:40:17 +0000"],"X-Ob_mode":["1"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Robots-Tag":["noindex, nofollow"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Encoding":["gzip"],"Vary":["Accept-Encoding"],"X-Xss-Protection":["1; mode=block"],"Content-Type":["text/html; charset=utf-8"]},"status":200}
{"level":"debug","ts":"2021-05-24T16:40:18.629+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/js/messages.php?l=en&v=5.1.0","headers":{"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Cf-Connecting-Ip":["49.196.36.201"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Cf-Ray":["654537ebdde41d83-NRT"],"Sec-Fetch-Mode":["no-cors"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cdn-Loop":["cloudflare"],"X-Forwarded-Proto":["https"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=5sh31f5b1gpecd8pbjg3fgj6ml; tk_or=%22%22; tk_lr=%22%22"],"Accept":["*/*"],"Sec-Fetch-Site":["same-origin"],"Cf-Request-Id":["0a3f21476900001d833ba68000000001"],"X-Forwarded-For":["49.196.36.201, 162.158.118.125, 10.1.1.4"],"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/js/messages.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"gzip","HTTP_ACCEPT_LANGUAGE":"en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"49.196.36.201","HTTP_CF_IPCOUNTRY":"AU","HTTP_CF_RAY":"654537ebdde41d83-NRT","HTTP_CF_REQUEST_ID":"0a3f21476900001d833ba68000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=5sh31f5b1gpecd8pbjg3fgj6ml; tk_or=%22%22; tk_lr=%22%22","HTTP_HOST":"test.lan:443","HTTP_SEC_FETCH_MODE":"no-cors","HTTP_SEC_FETCH_SITE":"same-origin","HTTP_USER_AGENT":"Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155","HTTP_X_FORWARDED_FOR":"49.196.36.201, 162.158.118.125, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"l=en&v=5.1.0","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"46355","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/js/messages.php?l=en&v=5.1.0","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/js/messages.php","SCRIPT_NAME":"/phpmyadmin/js/messages.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:40:18.823+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/js/messages.php?l=en&v=5.1.0","headers":{"Accept-Language":["en-AU,en-GB;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Host":["test.udance.com.au"],"Cf-Ipcountry":["AU"],"Cdn-Loop":["cloudflare"],"Accept-Encoding":["gzip"],"User-Agent":["Mozilla/5.0 (Linux; Android 11; Pixel 3 XL) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.116 Mobile Safari/537.36 EdgA/46.03.4.5155"],"Cf-Connecting-Ip":["49.196.36.201"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Cf-Ray":["654537ebdde41d83-NRT"],"Sec-Fetch-Mode":["no-cors"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"X-Forwarded-Proto":["https"],"Cf-Request-Id":["0a3f21476900001d833ba68000000001"],"Cookie":["pma_lang_https=en; phpMyAdmin_https=5sh31f5b1gpecd8pbjg3fgj6ml; tk_or=%22%22; tk_lr=%22%22"],"Accept":["*/*"],"Sec-Fetch-Site":["same-origin"],"X-Forwarded-For":["49.196.36.201, 162.158.118.125, 10.1.1.4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"Expires":["Mon, 24 May 2021 09:40:18 GMT"],"X-Ob_mode":["1"],"Content-Encoding":["gzip"],"Vary":["Accept-Encoding"],"X-Powered-By":["PHP/7.4.16"],"Content-Type":["text/javascript; charset=UTF-8"]},"status":200}
{"level":"debug","ts":"2021-05-24T16:40:30.783+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"HEAD","host":"test.lan:443","uri":"/","headers":{"X-Forwarded-Host":["test.udance.com.au"],"Cdn-Loop":["cloudflare"],"Cf-Request-Id":["0a3f2174c70000c7ea6b1c2000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["192.0.91.177, 172.69.71.67"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["654538347aefc7ea-DFW"],"Cf-Connecting-Ip":["192.0.91.177"],"X-Forwarded-Proto":["https"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Accept-Encoding":["gzip"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"HEAD","uri":"/index.php"}
{"level":"debug","ts":"2021-05-24T16:40:30.784+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"HEAD","host":"test.lan:443","uri":"/index.php","headers":{"Cf-Connecting-Ip":["192.0.91.177"],"X-Forwarded-Proto":["https"],"User-Agent":["jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)"],"Accept-Encoding":["gzip"],"X-Forwarded-Host":["test.udance.com.au"],"Cdn-Loop":["cloudflare"],"Cf-Request-Id":["0a3f2174c70000c7ea6b1c2000000001"],"Cf-Ipcountry":["US"],"X-Forwarded-For":["192.0.91.177, 172.69.71.67, 10.1.1.4"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"Cf-Ray":["654538347aefc7ea-DFW"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT_ENCODING":"gzip","HTTP_CDN_LOOP":"cloudflare","HTTP_CF_CONNECTING_IP":"192.0.91.177","HTTP_CF_IPCOUNTRY":"US","HTTP_CF_RAY":"654538347aefc7ea-DFW","HTTP_CF_REQUEST_ID":"0a3f2174c70000c7ea6b1c2000000001","HTTP_CF_VISITOR":"{\"scheme\":\"https\"}","HTTP_HOST":"test.lan:443","HTTP_USER_AGENT":"jetmon/1.0 (Jetpack Site Uptime Monitor by WordPress.com)","HTTP_X_FORWARDED_FOR":"192.0.91.177, 172.69.71.67, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"46355","REMOTE_USER":"","REQUEST_METHOD":"HEAD","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:40:32.903+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621845632.7852690219879150390625","headers":{"Accept-Encoding":["deflate, gzip"],"X-Forwarded-Proto":["https"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Content-Length":["0"],"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621845632.7852690219879150390625"],"X-Forwarded-Host":["test.udance.com.au"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"test.lan:443","HTTP_REFERER":"https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621845632.7852690219879150390625","HTTP_USER_AGENT":"WordPress/5.7.2; https://test.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1621845632.7852690219879150390625","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"46355","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1621845632.7852690219879150390625","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T16:40:33.025+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:46355","proto":"HTTP/2.0","method":"POST","host":"test.lan:443","uri":"/wp-cron.php?doing_wp_cron=1621845632.7852690219879150390625","headers":{"Content-Type":["application/x-www-form-urlencoded"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"],"Referer":["https://test.udance.com.au/wp-cron.php?doing_wp_cron=1621845632.7852690219879150390625"],"X-Forwarded-Host":["test.udance.com.au"],"Accept-Encoding":["deflate, gzip"],"X-Forwarded-Proto":["https"],"User-Agent":["WordPress/5.7.2; https://test.udance.com.au"],"Accept":["*/*"],"Content-Length":["0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Powered-By":["PHP/7.4.16"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"]},"status":200}

INTERNAL

Frontend

{"level":"debug","ts":"2021-05-24T17:00:29.085+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"{backend}","request":{"remote_addr":"10.1.1.222:50316","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"X-Forwarded-Proto":["https"],"Accept-Encoding":["gzip, deflate, br"],"X-Forwarded-For":["10.1.1.222"],"Sec-Fetch-User":["?1"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Sec-Fetch-Dest":["document"],"X-Forwarded-Host":["test.udance.com.au"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Cookie":["pma_lang_https=en; phpMyAdmin_https=8epmdea0qnpdtqt4tit0p72bao"],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"test.udance.com.au"}},"headers":{"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Content-Encoding":["gzip"],"Last-Modified":["Mon, 24 May 2021 09:00:29 +0000"],"Referrer-Policy":["no-referrer"],"Set-Cookie":["phpMyAdmin_https=rb045gnspcdqp6tmv6hvjc6jil; path=/phpmyadmin/; secure; HttpOnly"],"Vary":["Accept-Encoding"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Frame-Options":["DENY"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Content-Type":["text/html; charset=utf-8"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Powered-By":["PHP/7.4.16"],"X-Robots-Tag":["noindex, nofollow"],"Pragma":["no-cache"],"X-Content-Type-Options":["nosniff"],"Date":["Mon, 24 May 2021 09:00:29 GMT"],"Expires":["Mon, 24 May 2021 09:00:29 +0000"],"Server":["Caddy"],"X-Ob_mode":["1"],"X-Xss-Protection":["1; mode=block"]},"status":200}

Backend

{"level":"debug","ts":"2021-05-24T17:00:28.830+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:25917","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/","headers":{"Accept-Encoding":["gzip, deflate, br"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"],"X-Forwarded-Proto":["https"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-For":["10.1.1.222"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Cookie":["pma_lang_https=en; phpMyAdmin_https=8epmdea0qnpdtqt4tit0p72bao"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"method":"GET","uri":"/phpmyadmin/index.php"}
{"level":"debug","ts":"2021-05-24T17:00:28.830+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:25917","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-User":["?1"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Cookie":["pma_lang_https=en; phpMyAdmin_https=8epmdea0qnpdtqt4tit0p72bao"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"],"Accept-Encoding":["gzip, deflate, br"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/phpmyadmin/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_AUTHORIZATION":"Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ==","HTTP_COOKIE":"pma_lang_https=en; phpMyAdmin_https=8epmdea0qnpdtqt4tit0p72bao","HTTP_HOST":"test.lan:443","HTTP_SEC_CH_UA":"\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"test.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"25917","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/phpmyadmin/","SCRIPT_FILENAME":"/usr/local/www/wordpress/phpmyadmin/index.php","SCRIPT_NAME":"/phpmyadmin/index.php","SERVER_NAME":"test.lan","SERVER_PORT":"80","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.1","SSL_CIPHER":"TLS_CHACHA20_POLY1305_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-05-24T17:00:29.085+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:25917","proto":"HTTP/2.0","method":"GET","host":"test.lan:443","uri":"/phpmyadmin/index.php","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"X-Forwarded-Proto":["https"],"Accept-Language":["en-US,en;q=0.9"],"X-Forwarded-Host":["test.udance.com.au"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-User":["?1"],"Authorization":["Basic YWRtaW46OComQXZLTFJ0SCUyITJGaUFlZQ=="],"Cookie":["pma_lang_https=en; phpMyAdmin_https=8epmdea0qnpdtqt4tit0p72bao"],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"90\", \"Microsoft Edge\";v=\"90\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.212 Safari/537.36 Edg/90.0.818.66"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","proto_mutual":true,"server_name":"test.lan"}},"headers":{"X-Xss-Protection":["1; mode=block"],"Pragma":["no-cache"],"Content-Type":["text/html; charset=utf-8"],"X-Powered-By":["PHP/7.4.16"],"X-Content-Security-Policy":["default-src 'self' ;options inline-script eval-script;referrer no-referrer;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"Expires":["Mon, 24 May 2021 09:00:29 +0000"],"Last-Modified":["Mon, 24 May 2021 09:00:29 +0000"],"Content-Security-Policy":["default-src 'self' ;script-src 'self' 'unsafe-inline' 'unsafe-eval' ;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Content-Type-Options":["nosniff"],"Vary":["Accept-Encoding"],"X-Ob_mode":["1"],"X-Frame-Options":["DENY"],"X-Webkit-Csp":["default-src 'self' ;script-src 'self'  'unsafe-inline' 'unsafe-eval';referrer no-referrer;style-src 'self' 'unsafe-inline' ;img-src 'self' data:  *.tile.openstreetmap.org;object-src 'none';"],"X-Robots-Tag":["noindex, nofollow"],"Cache-Control":["no-store, no-cache, must-revalidate,  pre-check=0, post-check=0, max-age=0"],"Content-Encoding":["gzip"],"Set-Cookie":["phpMyAdmin_https=rb045gnspcdqp6tmv6hvjc6jil; path=/phpmyadmin/; secure; HttpOnly"],"Referrer-Policy":["no-referrer"]},"status":200}

So, I have some good news and some bad news. The good news is that Caddy mTLS works perfectly on FreeBSD. The bad news is that I haven’t been able to successfully roll it into the WordPress backend I have set up. I’ll summarise the good news in this post first and discuss the bad news in the next post.

Assumptions

The first assumption is that I have set up two FreeBSD jails; a frontend reverse proxy server in one jail and a web server in the second jail. Caddy is installed in both jails using the package manager pkg install caddy. At the time of preparing this post, I’m working with these versions:

# freebsd-version
12.2-RELEASE
# pkg info caddy
caddy-2.3.0_1

If using the DNS challenge, the frontend Caddy binary will need to be replaced using xcaddy to build a version of Caddy with a supported DNS provider module.

The second assumption this thread makes is that Caddy is serving subdomains of domain.com using a wildcard certificate. The map handler is used in the Caddyfile to facilitate managing the subdomains.

Local DNS resolver

The example below assumes acme.lan resolves to the frontend jail IP and test.lan resolves to backend jail IP.

Frontend jail considerations

Key frontend Caddyfile constructs required for mTLS:

...
# Internal CA
acme.lan {
  acme_server
  tls internal
}
...
*.domain.com {
  ...
  map {labels.2} {backend} {mtls} {

#   HOSTNAME     BACKEND         mTLS  #COMMENT
#---------------------------------------------------------------
  ...
    test         test.lan:443    yes   # test.domain.com
  ...
  route {
    ...
# Secure backend communication
    @mtls expression `{mtls} == "yes"`
    reverse_proxy @mtls {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
# Unsecured backend communication
    @nomtls expression `{mtls} == "no"`
    reverse_proxy @nomtls {backend}
    ...
  }
}

Next, and this is the secret ingredient to make mTLS work for FreeBSD, the root certificate for the internal CA has to be added to the system trust.

cat /var/db/caddy/data/caddy/pki/authorities/local/root.crt >> /usr/local/share/certs/ca-root-nss.crt

Limitations

The arrangement breaks if the ca_root_css package is upgraded in the frontend jail. When this happens, the local CA root certificate will have to be added to the system trust again.

Backend jail considerations

Key backend Caddyfile constructs required for mTLS:

{
  ...
  acme_ca https://acme.lan/acme/local/directory
  acme_ca_root /etc/ssl/certs/root.crt
}

test.lan {
  ...
}

Remember to add the local CA root certificate from the frontend to /etc/ssl/certs/ in the backend,

Next, for test.lan, I set up a static file server:

test.lan {
  root * /usr/local/www/caddy
  file server browse
}

I can reliably and consistently access the file server through test.domain.com. The Caddy reverse proxy provides automatic HTTPS, and mTLS ensures that the path between the frontend and backend Caddy servers is encrypted.

2 Likes

Now the bad news…

WordPress is a bit temperamental when it is hosted behind a reverse proxy that provides TLS, but local traffic behind the reverse proxy is unencrypted. so WordPress itself is hosted without TLS. In this situation, the following code has to be added to the WordPress configuration file wp-config.php to prevent an infinite redirect loop.

if (strpos($_SERVER['HTTP_X_FORWARDED_PROTO'], 'https') !== false)
$_SERVER['HTTPS']='on';

Reference: WordPress: Using a Reverse Proxy

When a reverse proxy is not involved, though I haven’t tried it myself, the WordPress documentation suggests that setting up WordPress with HTTPS is straightforward.

Reference: WordPress: HTTPS for WordPress.

What I have now is not covered by either of the above scenarios i.e. WordPress behind a reverse proxy, but with the backend retrospectively being encrypted using mTLS. More recent posts in this thread suggest that WordPress is responding to mTLS, but is now one step removed from the frontend and can’t seem to find its way back there.

I’ve dropped this in the lap of WordPress support to see if they have any ideas. You can follow the thread here mTLS and WordPress.

This topic was automatically closed after 30 days. New replies are no longer allowed.