WordPress and mTLS

basil · July 12, 2021, 8:42am

1. Caddy version (`caddy version`):

2. How I run Caddy:

Frontend Caddy reverse proxy server built with the Cloudflare module

root@caddy:~ # caddy version
v2.4.4-0.20210621175641-2de7e14e1c5f h1:/Kzlg8YluMMiXJBPoL8MkmArv5yqieoLHqKUDNuHtjE=

Backend WordPress PHP web server using a static binary.

root@wp-mockup:/usr/local/www # caddy version
v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

a. System environment:

root@caddy:~ # freebsd-version
12.2-RELEASE-p6

b. Command:

service caddy start

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

Relevant extract for the Caddyfile for the frontend reverse proxy server and internal CA

...
(proxy-mtls) {
  @{args.0} host {args.0}.udance.com.au
  reverse_proxy @{args.0} https://{args.1} {
    header_up Host {http.reverse_proxy.upstream.hostport}
    header_up X-Forwarded-Host {host}
  }
}
...
acme.lan {
  acme_server
  tls internal
}
...
*.udance.com.au {
  ...
  import proxy-mtls  mockup  mockup.lan
  ...
}

Relevant extract for the backend WordPress PHP file server:

{
  ...
  acme_ca https://acme.lan/acme/local/directory
  acme_ca_root /etc/ssl/certs/root.crt
  debug
}
mockup.lan {
  ...
  root * /usr/local/www/wordpress
  php_fastcgi 127.0.0.1:9000 {
    env SERVER_PORT 80 # Needed for Jetpack. With mTLS, I may need to change this to 443.
  }
  file_server
  ...
}

3. The problem I’m having:

There have been two issues that have inhibited me from fully embracing mTLS. The first of these had to do with mTLS stability. This looks like it may have resolved itself already (see mTLS: tls internal error - #19 by basil). I’m monitoring this closely.

The second issue has to do with getting WordPress to work with mTLS. For me, it’s a stumbling block to the wider adoption of mTLS. I have visited this previously and made some comments (starting with this post mTLS under FreeBSD - #65 by basil). I’m revisiting the issue in this thread in the hope that a fresh perspective will generate some new ideas that may lead to a resolution. See the posts that follow.

4. Error messages and/or full log output:

See posts to follow.

5. What I already tried:

See links in the next section.

6. Links to relevant resources:

basil · July 12, 2021, 9:17am

The diagram below highlights the important pieces required to expose a WordPress site to the internet. Caddy manages the bits in red. Without the Caddyfile code snippets, my WordPress instance would remain locked behind the local network and never be exposed to the internet. It’s the magic that makes this a reality.

The WordPress PHP code snippet probably requires some explanation. It’s derived from two sources. The first source has to do with getting WordPress to work behind a reverse proxy (see WordPress: Using a Reverse Proxy). The second source has to do with getting WP-CLI, the command-line interface to WordPress, to do the same (see PHP notice: Undefined index on $_SERVER superglobal)

This structural arrangement has been ‘soak tested’ over a lengthy period and I consider it a trusted and robust approach to making WordPress accessible and secure beyond the local network. What I’d like to do now though is have Caddy manage the unencrypted path between the file server and the reverse proxy. My Caddy life would then be complete

basil · July 12, 2021, 9:38am

However, when I enable mTLS, the arrangement breaks. In the diagram below, the code in the mango colour is what’s required to enable mTLS.

Debug log extracts to follow in the next post.

basil · July 12, 2021, 10:57am

When I first attempt to access the WordPress site mockup.udance.com.au, this is what I see in the browser…

It’s curious how mockup.lan gets revealed in the browser.

This is what appears in the frontend Caddy process log…

{"level":"debug","ts":"2021-07-12T18:21:27.226+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mockup.lan:443","request":{"remote_addr":"10.1.1.50:49990","proto":"HTTP/1.1","method":"POST","host":"mockup.lan:443","uri":"/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000","headers":{"X-Forwarded-For":["10.1.1.50"],"Accept-Encoding":["deflate, gzip"],"Referer":["https://mockup.udance.com.au/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000"],"X-Forwarded-Proto":["https"],"Content-Length":["0"],"X-Forwarded-Host":["mockup.udance.com.au"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://mockup.udance.com.au"],"Accept":["*/*"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"http/1.1","proto_mutual":true,"server_name":"mockup.udance.com.au"}},"headers":{"X-Powered-By":["PHP/7.4.20"],"Content-Length":["0"],"Date":["Mon, 12 Jul 2021 10:21:27 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Server":["Caddy"]},"status":200}
{"level":"debug","ts":"2021-07-12T18:21:27.231+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"mockup.lan:443","request":{"remote_addr":"10.1.1.222:55880","proto":"HTTP/2.0","method":"GET","host":"mockup.lan:443","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"X-Forwarded-For":["10.1.1.222"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"X-Forwarded-Host":["mockup.udance.com.au"],"Sec-Fetch-Site":["none"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"mockup.udance.com.au"}},"headers":{"Content-Type":["text/html; charset=UTF-8"],"Location":["https://mockup.lan/"],"Server":["Caddy"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.20"],"X-Redirect-By":["WordPress"],"Content-Length":["0"],"Date":["Mon, 12 Jul 2021 10:21:27 GMT"]},"status":301}

This is what appears in the backend Caddy process log…

{"level":"debug","ts":"2021-07-12T18:21:26.987+0800","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_addr":"10.1.1.4:25817","proto":"HTTP/2.0","method":"GET","host":"mockup.lan:443","uri":"/","headers":{"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"X-Forwarded-For":["10.1.1.222"],"Sec-Fetch-Dest":["document"],"X-Forwarded-Host":["mockup.udance.com.au"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"Sec-Ch-Ua-Mobile":["?0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"mockup.lan"}},"method":"GET","uri":"/index.php"}
{"level":"debug","ts":"2021-07-12T18:21:26.987+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:25817","proto":"HTTP/2.0","method":"GET","host":"mockup.lan:443","uri":"/index.php","headers":{"X-Forwarded-Host":["mockup.udance.com.au"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Fetch-Dest":["document"],"Sec-Ch-Ua-Mobile":["?0"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Mode":["navigate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"mockup.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"","CONTENT_TYPE":"","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/index.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9","HTTP_ACCEPT_ENCODING":"gzip, deflate, br","HTTP_ACCEPT_LANGUAGE":"en-US,en;q=0.9","HTTP_HOST":"mockup.lan:443","HTTP_SEC_CH_UA":"\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\"","HTTP_SEC_CH_UA_MOBILE":"?0","HTTP_SEC_FETCH_DEST":"document","HTTP_SEC_FETCH_MODE":"navigate","HTTP_SEC_FETCH_SITE":"none","HTTP_SEC_FETCH_USER":"?1","HTTP_UPGRADE_INSECURE_REQUESTS":"1","HTTP_USER_AGENT":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67","HTTP_X_FORWARDED_FOR":"10.1.1.222, 10.1.1.4","HTTP_X_FORWARDED_HOST":"mockup.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"25817","REMOTE_USER":"","REQUEST_METHOD":"GET","REQUEST_SCHEME":"https","REQUEST_URI":"/","SCRIPT_FILENAME":"/usr/local/www/wordpress/index.php","SCRIPT_NAME":"/index.php","SERVER_NAME":"mockup.lan","SERVER_PORT":"443","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.3","SSL_CIPHER":"TLS_AES_128_GCM_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-07-12T18:21:27.225+0800","logger":"http.reverse_proxy.transport.fastcgi","msg":"roundtrip","request":{"remote_addr":"10.1.1.4:25817","proto":"HTTP/2.0","method":"POST","host":"mockup.lan:443","uri":"/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000","headers":{"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://mockup.udance.com.au"],"Accept":["*/*"],"Referer":["https://mockup.udance.com.au/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000"],"Accept-Encoding":["deflate, gzip"],"Content-Length":["0"],"X-Forwarded-Host":["mockup.udance.com.au"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"mockup.lan"}},"dial":"127.0.0.1:9000","env":{"AUTH_TYPE":"","CONTENT_LENGTH":"0","CONTENT_TYPE":"application/x-www-form-urlencoded","DOCUMENT_ROOT":"/usr/local/www/wordpress","DOCUMENT_URI":"/wp-cron.php","GATEWAY_INTERFACE":"CGI/1.1","HTTPS":"on","HTTP_ACCEPT":"*/*","HTTP_ACCEPT_ENCODING":"deflate, gzip","HTTP_CONTENT_LENGTH":"0","HTTP_CONTENT_TYPE":"application/x-www-form-urlencoded","HTTP_HOST":"mockup.lan:443","HTTP_REFERER":"https://mockup.udance.com.au/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000","HTTP_USER_AGENT":"WordPress/5.7.2; https://mockup.udance.com.au","HTTP_X_FORWARDED_FOR":"10.1.1.50, 10.1.1.4","HTTP_X_FORWARDED_HOST":"mockup.udance.com.au","HTTP_X_FORWARDED_PROTO":"https","PATH_INFO":"","QUERY_STRING":"doing_wp_cron=1626085287.1896800994873046875000","REMOTE_ADDR":"10.1.1.4","REMOTE_HOST":"10.1.1.4","REMOTE_IDENT":"","REMOTE_PORT":"25817","REMOTE_USER":"","REQUEST_METHOD":"POST","REQUEST_SCHEME":"https","REQUEST_URI":"/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000","SCRIPT_FILENAME":"/usr/local/www/wordpress/wp-cron.php","SCRIPT_NAME":"/wp-cron.php","SERVER_NAME":"mockup.lan","SERVER_PORT":"443","SERVER_PROTOCOL":"HTTP/2.0","SERVER_SOFTWARE":"Caddy/v2.4.3","SSL_CIPHER":"TLS_AES_128_GCM_SHA256","SSL_PROTOCOL":"TLSv1.3"}}
{"level":"debug","ts":"2021-07-12T18:21:27.226+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:25817","proto":"HTTP/2.0","method":"POST","host":"mockup.lan:443","uri":"/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000","headers":{"Accept-Encoding":["deflate, gzip"],"Content-Length":["0"],"Content-Type":["application/x-www-form-urlencoded"],"User-Agent":["WordPress/5.7.2; https://mockup.udance.com.au"],"Accept":["*/*"],"Referer":["https://mockup.udance.com.au/wp-cron.php?doing_wp_cron=1626085287.1896800994873046875000"],"X-Forwarded-Host":["mockup.udance.com.au"],"X-Forwarded-Proto":["https"],"X-Forwarded-For":["10.1.1.50, 10.1.1.4"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"mockup.lan"}},"headers":{"Expires":["Wed, 11 Jan 1984 05:00:00 GMT"],"Cache-Control":["no-cache, must-revalidate, max-age=0"],"Content-Type":["text/html; charset=UTF-8"],"X-Powered-By":["PHP/7.4.20"]},"status":200}
{"level":"debug","ts":"2021-07-12T18:21:27.230+0800","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"127.0.0.1:9000","request":{"remote_addr":"10.1.1.4:25817","proto":"HTTP/2.0","method":"GET","host":"mockup.lan:443","uri":"/index.php","headers":{"Sec-Fetch-User":["?1"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Mode":["navigate"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36 Edg/91.0.864.67"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Dest":["document"],"X-Forwarded-Host":["mockup.udance.com.au"],"Sec-Ch-Ua":["\" Not;A Brand\";v=\"99\", \"Microsoft Edge\";v=\"91\", \"Chromium\";v=\"91\""],"X-Forwarded-For":["10.1.1.222, 10.1.1.4"],"Sec-Ch-Ua-Mobile":["?0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"mockup.lan"}},"headers":{"X-Redirect-By":["WordPress"],"Location":["https://mockup.lan/"],"Status":["301 Moved Permanently"],"X-Powered-By":["PHP/7.4.20"],"Content-Type":["text/html; charset=UTF-8"]},"status":301}
{"level":"debug","ts":"2021-07-12T18:21:27.258+0800","logger":"http.stdlib","msg":"http: TLS handshake error from 10.1.1.222:57147: remote error: tls: unknown certificate"}

Somehow,mockup.udance.com.au has got ‘lost in the wilderness’ and can’t seem to find its way back to the client. I’m hoping there’s a magic tweak to fix this

francislavoie · July 12, 2021, 2:27pm

Sounds like Wordpress doesn’t know how to use X-Forwarded-Host.

You might need to override some env under php_fastcgi to override SERVER_NAME and/or HTTP_HOST I think (which are the ones which have the request’s Host header in them)

Or maybe you can add something like this in your wp-config.php, idk:

if (isset($_SERVER['HTTP_X_FORWARDED_HOST'])) {
    $_SERVER['HTTP_HOST'] = $_SERVER['HTTP_X_FORWARDED_HOST'];
}

basil · July 12, 2021, 5:01pm

francislavoie:

Or maybe you can add something like this in your wp-config.php, idk:
if (isset($_SERVER['HTTP_X_FORWARDED_HOST'])) {
    $_SERVER['HTTP_HOST'] = $_SERVER['HTTP_X_FORWARDED_HOST'];
}

OMG! That was pure genius!!!

I also noticed that SERVER_NAME was set to mockup.lan so I’ve updated that as well…

if (isset($_SERVER['HTTP_X_FORWARDED_HOST'])) {
    $_SERVER['HTTP_HOST'] = $_SERVER['HTTP_X_FORWARDED_HOST'];
    $_SERVER['SERVER_NAME'] = $_SERVER['HTTP_X_FORWARDED_HOST'];
}

Thank you! Thank you! Thank you! This had me stumped.

The diagram below shows what has to be included (mango colour) to enable mTLS when the upstream server is WordPress.

What is truly astounding is just how versatile Caddy is:

It’s the webserver serving WordPress PHP files;
It’s encrypting communications between the webserver and reverse proxy;
It’s providing reverse proxy services; and
automatic HTTPS.

francislavoie · July 13, 2021, 4:59pm

Btw I just want to make sure it’s said – don’t use that chunk of code in wp-config.php in installations where requests may come from untrusted clients (i.e. public WordPress instances with no Caddy in front, or instances of Caddy where you didn’t use header_up X-Forwarded-Host) because then some bad actors could send that header in the request to possibly do bad things (the effect would probably be that they redirect themselves to whatever domain they put in that header, but I don’t trust WordPress’ terrible codebase to not have some other terrible side effect).

basil · July 13, 2021, 5:11pm

I was blissfully unaware of that. Thanks for the heads-up.

basil · August 1, 2021, 3:55am

WordPress and mTLS has been rock solid over the month of July since implementing the ‘magic’ WP code. When I get around to it, I’ll wikify this thread and then lobby to have the wiki article referenced somewhere in the WordPress support article Administration Over SSL. This will be a unique reference as I’ve not come across evidence of historically popular proxy servers being configured to include mTLS when WordPress is behind a reverse proxy.

system · August 11, 2021, 8:43am

This topic was automatically closed after 30 days. New replies are no longer allowed.