mTLS under FreeBSD

[basil (Basil Hendroff)]

1. Caddy version (caddy version):

Frontend Caddy reverse proxy server built with the Cloudflare module

root@caddy:~ # caddy version
v2.4.0-rc.1 h1:tZl6bDhlwtRwuWpebRUYpDJPhJaGyrXIMp7fmuMXwMc=

Backend Caddy web server using a static binary

root@wordpress:~ # caddy version
v2.4.0-rc.1 h1:tZl6bDhlwtRwuWpebRUYpDJPhJaGyrXIMp7fmuMXwMc=

2. How I run Caddy:

a. System environment:

root@caddy:~ # freebsd-version
12.2-RELEASE-p6

b. Command:

service caddy start

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

Caddyfile extract with the key elements being a wildcard domain and the map directive.

*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE PHPMYADMIN
#---------------------------------------------------------------
    ...
    # Jails
    ...
    test         10.1.1.50:80    yes    yes      # test.udance.com.au
    ...
  }
  ...
  reverse_proxy {backend}
}

3. The problem I’m having:

None at this stage. This is really a journal of my personal journey to get mTLS working between a frontend Caddy reverse_proxy server and a backend Caddy webserver. I’ve no doubt there’ll be stumbling blocks along the way.

4. Error messages and/or full log output:

None at this stage. This OP just lays out the groundwork i.e. a snapshot of my starting position.

5. What I already tried:

The links in the next section ‘line up the ducks’ for this exercise.

6. Links to relevant resources:

1. Forum thread Reverse proxy parsing difficulties
2. Wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts
3. Forum thread Migrate to using a wildcard certificate
4. Wiki article Using Caddy for incident management in a home network

[basil (Basil Hendroff)]

First step was to set up the ACME server in the frontend Caddyfile.

# ACME server
caddy.lan {
  acme_server
  tls internal
}

My first stumbling block. … I reload Caddy. Checking the log, I note errors around NSS support:

{"level":"warn","ts":"2021-05-04T14:29:41.537+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-04T14:29:42.094+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00010a230"}
{"level":"info","ts":"2021-05-04T14:29:42.096+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-04T14:29:42.096+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620109782.231529,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620109782.2527225,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-04T14:29:42.263+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-04T14:29:42.264+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003bdf80"}
{"level":"info","ts":"2021-05-04T14:29:42.296+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-04T14:29:42.296+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-04T14:29:56.456+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["caffigoalkeeping.com.au","caffigoalkeeping.com","www.readymcgetty.com.au","readymcgetty.com.au","www.udance.com.au","caddy.lan","www.caffigoalkeeping.com.au","udance.com.au","*.udance.com.au","www.caffigoalkeeping.com","www.xenografix.com.au","xenografix.com.au"]}
{"level":"warn","ts":"2021-05-04T14:29:56.478+0800","logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [caddy.lan]: no OCSP server specified in certificate"}
{"level":"warn","ts":"2021-05-04T14:29:56.539+0800","logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
2021/05/04 14:29:56 define JAVA_HOME environment variable to use the Java trust
2021/05/04 14:29:56 Note: NSS support is not available on your platform
{"level":"error","ts":"2021-05-04T14:29:56.539+0800","logger":"pki.ca.local","msg":"failed to install root certificate","error":"trust not supported","certificate_file":"storage:pki/authorities/local/root.crt"}
{"level":"info","ts":"2021-05-04T14:29:56.539+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-04T14:29:56.540+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-04T14:29:56.540+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=30573) - Caddy is running in the background

Is there a package dependency here? Maybe I need to install additional packages to support mTLS, though I’m not sure what they might be?

Just to be sure the lines added to the Caddyfile were causing the issue, I temporarily commented them out and restarted Caddy. The log shows no errors.

{"level":"info","ts":"2021-05-04T18:19:22.511+0800","msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":"2021-05-04T18:19:22.511+0800","msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":"2021-05-04T18:19:22.794+0800","logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0001e0700"}
{"level":"info","ts":"2021-05-04T18:19:22.796+0800","logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":"2021-05-04T18:19:22.796+0800","msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
{"level":"info","ts":1620123562.9296148,"msg":"using provided configuration","config_file":"/usr/local/www/Caddyfile","config_adapter":"caddyfile"}
{"level":"warn","ts":1620123562.9503217,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/usr/local/www/Caddyfile","line":2}
{"level":"info","ts":"2021-05-04T18:19:22.958+0800","logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
{"level":"info","ts":"2021-05-04T18:19:22.959+0800","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0003b2620"}
{"level":"info","ts":"2021-05-04T18:19:22.960+0800","logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
{"level":"info","ts":"2021-05-04T18:19:22.960+0800","logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
{"level":"info","ts":"2021-05-04T18:19:36.410+0800","logger":"http","msg":"enabling automatic TLS certificate management","domains":["www.xenografix.com.au","readymcgetty.com.au","www.caffigoalkeeping.com","xenografix.com.au","caffigoalkeeping.com","www.readymcgetty.com.au","caffigoalkeeping.com.au","udance.com.au","*.udance.com.au","www.caffigoalkeeping.com.au","www.udance.com.au"]}
{"level":"info","ts":"2021-05-04T18:19:36.409+0800","logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/.local/share/caddy"}
{"level":"info","ts":"2021-05-04T18:19:36.438+0800","logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":"2021-05-04T18:19:36.457+0800","msg":"autosaved config (load with --resume flag)","file":"/.config/caddy/autosave.json"}
{"level":"info","ts":"2021-05-04T18:19:36.457+0800","msg":"serving initial configuration"}
Successfully started Caddy (pid=68418) - Caddy is running in the background

[Whitestrake (Matthew Fay)]

I believe NSS refers to Mozilla’s Network Security Services libraries?

You can either build and install them for your platform, or you could just grab the root certificate from the location noted in the logs ("storage:pki/authorities/local/root.crt", where I understand storage should be the data directory) and whack that wherever your trust store is.

[francislavoie]

You can ignore that message entirely if you don’t use Firefox on that machine. That’s just the underlying smallstep libs trying to install the root cert for Firefox since it doesn’t use the system’s trust store.

[basil (Basil Hendroff)]

@Whitestrake @francislavoie Phew! I was beginning to think that was the end of my mTLS journey

@Whitestrake

where I understand storage should be the data directory) and whack that wherever your trust store is.

For a start, I had no idea where that would be.

I also spotted this thread that had me worried as it referred to FreeBSD Starting with caddy2 - basic Caddyfile trying to use port 80

@francislavoie

You can ignore that message entirely if you don’t use Firefox on that machine. That’s just the underlying smallstep libs trying to install the root cert for Firefox since it doesn’t use the system’s trust store.

Thank goodness!

[basil (Basil Hendroff)]

As I want to be selective about which backend services should have secured comms, I altered the map table and added a couple of handles in the wildcard domain Caddy block to give me that flexibility.

*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
    ...
    # Jails
    ...
    test         10.1.1.50:80    yes    yes  yes        # test.udance.com.au

    ...
  }
  ...
# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    # Watch this space!
  }

# Unsecured backend communication

  @nomtls expression `{mtls} == "no"`
  handle @nomtls {
    reverse_proxy {backend}
  }
}

Jump in anytime if you think there is a better way of going about this.

[basil (Basil Hendroff)]

I should have tested out the modified wildcard domain before continuing. It turns out it wasn’t working the way I expected it to. Attempting to access a subdomain (when {mtls} is either yes and no} returned a blank screen.

An extract from caddy adapt --pretty for the unmodified wildcard domain block…

                                                                                {
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "reverse_proxy",
                                                                                                        "upstreams": [
                                                                                                                {
                                                                                                                        "dial": "{backend}"
                                                                                                                }
                                                                                                        ]
                                                                                                }
                                                                                        ]
                                                                                }

…and from the modified wildcard domain block…

                                                                                {
                                                                                        "group": "group28",
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "subroute"
                                                                                                }
                                                                                        ],
                                                                                        "match": [
                                                                                                {
                                                                                                        "expression": "{mtls} == \"yes\""
                                                                                                }
                                                                                        ]
                                                                                },
                                                                                {
                                                                                        "group": "group28",
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "subroute",
                                                                                                        "routes": [
                                                                                                                {
                                                                                                                        "handle": [
                                                                                                                                {
                                                                                                                                        "handler": "reverse_proxy",
                                                                                                                                        "upstreams": [
                                                                                                                                                {
                                                                                                                                                        "dial": "{backend}"
                                                                                                                                                }
                                                                                                                                        ]
                                                                                                                                }
                                                                                                                        ]
                                                                                                                }
                                                                                                        ]
                                                                                                }
                                                                                        ],
                                                                                        "match": [
                                                                                                {
                                                                                                        "expression": "{mtls} == \"no\""
                                                                                                }
                                                                                        ]

It appears the handler changed from reverse_proxy to subroute with the modified wildcard subdomain. After reviewing @matt’s wiki article Composing in the Caddyfile and with a bit of experimentation, I found that wrapping the matcher and associated handle directive in a route block fixed the issue.

This was confirmed in this caddy adapt --pretty extract;

                          "handle": [
                                  {
                                          "handler": "subroute",
                                          "routes": [
                                                  {
                                                          "handle": [
                                                                  {
                                                                          "handler": "subroute"
                                                                  }
                                                          ],
                                                          "match": [
                                                                  {
                                                                          "expression": "{mtls} == \"yes\""
                                                                  }
                                                          ]
                                                  }
                                          ]
                                  },
                                  {
                                          "handler": "subroute",
                                          "routes": [
                                                  {
                                                          "handle": [
                                                                  {
                                                                          "handler": "subroute",
                                                                          "routes": [
                                                                                  {
                                                                                          "handle": [
                                                                                                  {
                                                                                                          "handler": "reverse_proxy",
                                                                                                          "upstreams": [
                                                                                                                  {
                                                                                                                          "dial": "{backend}"
                                                                                                                  }
                                                                                                          ]
                                                                                                  }
                                                                                          ]
                                                                                  }
                                                                          ]
                                                                  }
                                                          ],
                                                          "match": [
                                                                  {
                                                                          "expression": "{mtls} == \"no\""
                                                                  }
                                                          ]
                                                  }
                                          ]
                                  }
                          ]

This is the working wildcard subdomain Caddy block…

*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
    ...
    # Jails
    ...
    test         10.1.1.50:80    yes    yes  yes        # test.udance.com.au

    ...
  }
  ...
# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      # Watch this space!
    }
  }

# Unsecured backend communication

  route {
    @nomtls expression `{mtls} == "no"`
    handle @nomtls {
      reverse_proxy {backend}
    }
  }
}

[basil (Basil Hendroff)]

@basil

# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      # Watch this space!
    }
  }

So [head scratching], what do I replace ‘# Watch this space!’ with?

From @Rob789’s inspirational wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts , a quote from the section Local (Split) DNS…

I created domain names for the hosts because pointing directly to the IP addresses didn’t work with the ACME server. I was not able to get certificates for the backends.

So, to set the scene, my local DNS resolver (DNSMasq if anyone is interested) resolves:

1. caddy.lan → 10.1.1.4
2. test.lan → 10.1.1.50

…and this extract from the section Local HTTPS…

Secondly, you need define or update the FQDN where Caddy listens to and reverse proxies accordingly with TLS.

nextcloud.my.domain.com {
         reverse_proxy https://office.roadrunner {
              header_up Host {http.reverse_proxy.upstream.hostport}
              header_up X-Forwarded-Host {host}
         }
}

If I wasn’t using the map directive, the equivalent Caddy block for me would be:

test.udance.com.au {
         reverse_proxy https://test.lan {
              header_up Host {http.reverse_proxy.upstream.hostport}
              header_up X-Forwarded-Host {host}
         }
}

Using the map directive, here’s my first attempt on paper:

# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
      }
    }
  }

I’m not sure that the translation is correct? For a start, {backend} refers to an IP address and port number rather than a local domain name as suggested in the wiki article. Thoughts?

[francislavoie]

Almost. You need to enable HTTPS, and because of the other topic where I explained all that, you can’t use the scheme, so you have to use the “long way” of configuring the transport:

    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
        transport http {
          tls
        }
      }
    }

Also you don’t need the route wrapping, I don’t see what that gives you here. FYI both handle and route directives generate subroutes, it’s just that they have different effects on how the Caddyfile is parsed (route overrides the directive order – not useful here), and whether they become mutually exclusive or not (i.e. handle).

[Rob789 (Robbert)]

Hi Basil,

Good luck with your journey. I will try to follow it and help where I can.

Just one warning. My setup is not working 100%. There is something going wrong with the renewal of the certificates. They get renewed but I cannot establish a connection between the nodes anymore. I have to stop all Caddy services, delete the issued certificates in the upstream node and restart everything again.

I will most likely open a new post topic for this as soon as I get some time to look into this more.

[basil (Basil Hendroff)]

@francislavoie

Also you don’t need the route wrapping, I don’t see what that gives you here. FYI both handle and route directives generate subroutes, it’s just that they have different effects on how the Caddyfile is parsed (route overrides the directive order – not useful here), and whether they become mutually exclusive or not (i.e. handle).

Curious. Referring to post #7 above, without the route wrapper, a browser (both Google Chrome and Microsoft Edge tested) returns nothing back. To demonstrate what I’m seeing now:

I start with this…

    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse proxy {backend}
    }

From a browser, when I try to access test.udance.com.au, an empty screen is returned.

If I reinstate the route wrapper…

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse proxy {backend}
    }
  }

From a browser, I can now access the site test.udance.com.au

If I now add in the code to support mTLS…

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
        transport http {
          tls
        }
      }
    }
  }

From a browser, I get a 502 error returned. I’m not sure if this is an expected result at this stage as I haven’t added anything yet to the backend Caddyfile. Note: Removing the route wrapper, but leaving the additional code returns an empty screen again.

[francislavoie]

Oh if your map still has :80, then it needs to be :443, cause HTTPS is over port 443.

Re route, there must be something else going on in your config, cause there’s no reason for that to be the case.

Also to make reading the caddy adapt --pretty output less annoying, you can enter tabs -2 to set the tab size to 2 spaces for your current shell session.

If I write a Caddyfile like this:

*.udance.com.au {
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
    # Jails
    test         10.1.1.50:80    yes    yes  yes        # test.udance.com.au
  }
# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    reverse_proxy {backend}
  }

# Unsecured backend communication

  @nomtls expression `{mtls} == "no"`
  handle @nomtls {
    reverse_proxy {backend}
  }
}

Then adapt it, I get this, which all makes sense (subroutes in the right place etc)

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "*.udance.com.au"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "destinations": [
                            "{backend}",
                            "{online}",
                            "{mtls}",
                            "{phpmyadmin}"
                          ],
                          "handler": "map",
                          "mappings": [
                            {
                              "input": "test",
                              "outputs": [
                                "10.1.1.50:80",
                                "yes",
                                "yes",
                                "yes"
                              ]
                            }
                          ],
                          "source": "{http.request.host.labels.3}"
                        }
                      ]
                    },
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "{backend}"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "expression": "{mtls} == \"yes\""
                        }
                      ]
                    },
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "{backend}"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "expression": "{mtls} == \"no\""
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

(Also you could pipe the output into jq, like caddy adapt | jq which will syntax highlight it and format it. See jq )

[basil (Basil Hendroff)]

@francislavoie

Oh if your map still has :80, then it needs to be :443, cause HTTPS is over port 443.

Changed that. Still getting the 502 error.

@francislavoie

Also to make reading the caddy adapt --pretty output less annoying, you can enter tabs -2 to set the tab size to 2 spaces for your current shell session.

Thanks for the tip

@francislavoie

(Also you could pipe the output into jq, like caddy adapt | jq which will syntax highlight it and format it. See jq )

Also, good to know.

@francislavoie

Re route, there must be something else going on in your config, cause there’s no reason for that to be the case.

I had hoped to only highlight the minimal parts of my Caddyfile that are relevant to solving the mTLS puzzle, but for the sake of completeness, here are the relevant parts of my Caddyfile, including the global section and snippets, which relate to the udance.com.au domain.

{
  email basil.hendroff@udance.com.au
  acme_dns cloudflare [REDACTED]
#  debug
  log {
    format json {
      time_format iso8601
    }
  }
}

#----------------------------------------------------------------------
# Snippet   : Description                           : Arguments
#----------------------------------------------------------------------
# authorise : Basic authentication                  : subdirectory
# logging   : Rolling access log                    : subdomain
# online    : Domain availability                   : {yes|no|split}

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

(logging) {
  log {
    format json {
      time_format iso8601
    }
    output file /var/log/caddy/{args.0}.log {
      roll_size 100MiB    # Default 100MiB
      roll_keep 10        # Default 10
      roll_keep_for 90d   # Default 90d
    }
  }
}

www.udance.com.au {
  redir https://udance.com.au{uri} permanent
}

udance.com.au {

  encode gzip zstd
  import logging udance.com.au
  import authorise /phpmyadmin*

  map {path} {backend} {online} {

#   PATH                BACKEND          ONLINE
#---------------------------------------------------------------
    ~^/tautulli.*       10.1.1.26:8181   yes
    ~^/transmission.*   10.1.1.28:9091   yes
    ~^/.*               10.1.1.55:80     yes
  }

# Offline handling

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  handle @split {
    redir https://udance.statuspage.io temporary
  }

  reverse_proxy {backend}
}


*.udance.com.au {

  encode gzip zstd
  import logging udance.com.au

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    # Docker containers

    office       10.1.1.13:8880  yes    no   no         # OnlyOffice
    portainer    10.1.1.13:9000  yes    no   no         # Portainer
    truecommand  10.1.1.13:8080  yes    no   no         # TrueCommand
    tc123        10.1.1.13:8082  yes    no   no         # TrueCommand v1.2.3
    nc-fpm       10.1.1.13:8031  yes    no   no         # Nextcloud+Caddy
    wordpress    10.1.1.13:5050  yes    no   no         # WordPress
    nc-apache    10.1.1.13:8030  yes    no   no         # Nextcloud+Apache
    collabora    10.1.1.13:9980  yes    no   no         # Collabora

    # Jails

    rslsync      10.1.1.22:8888  yes    no   no         # Resilio Sync
    cloud        10.1.1.29:80    yes    no   no         # Nextcloud
    heimdall     10.1.1.23:80    yes    no   no         # Heimdall
    blog         10.1.1.54:80    yes    no   yes        # blog.udance.com.au
    test         10.1.1.50:443   yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling

  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  handle @split {
    redir https://udance.statuspage.io temporary
  }

# Authenticate phpMyAdmin on production WordPress sites

  @phpmyadmin expression `{phpmyadmin} == "yes"`
  handle @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.

  @nc-apache host nc-apache.udance.com.au
  handle @nc-apache {
    redir /.well-known/carddav /remote.php/carddav 301
    redir /.well-known/caldav /remote.php/caldav 301
  }

# Enable HSTS for Nextcloud

  @hsts host cloud.udance.com.au
  handle @hsts {
    header {
      Strict-Transport-Security max-age=31536000;
    }
  }

# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
        transport http {
          tls
        }
      }
    }
  }

# Unsecured backend communication

  route {
    @nomtls expression `{mtls} == "no"`
    handle @nomtls {
      reverse_proxy {backend}
    }
  }
}

[francislavoie]

Ah, it’s cause handle is mutually exclusive, so since you’re using that all over the place beforehand, it only ever runs the first matching handle block. Most of those should be changed to route instead probably since you do want them to fall through (like the ones for headers and basicauth). Specifically I think cause you set phpmyadmin to yes for that domain, it hits that handle then never runs the one with mtls.

Also I think you probably want to use the actual domain instead of the IP address for that one (and its domain needs to resolve to that IP address with your internal DNS server). That’s probably why it doesn’t work – your backend Caddy can’t complete the TLS handshake because it’s expecting to see a domain in the handshake but you’re just giving it an IP address and it doesn’t have a certificate for that IP address. I think. But check your logs.

[basil (Basil Hendroff)]

@francislavoie

Most of those should be changed to route instead probably since you do want them to fall through (like the ones for headers and basicauth).

Please correct me if I’m wrong, but I’m interpreting this as 'add a route wrapper for all the surrounding handle checks`.

@francislavoie

Specifically I think cause you set phpmyadmin to yes for that domain, it hits that handle then never runs the one with mtls.

Oh, I see.

@francislavoie

Also I think you probably want to use the actual domain instead of the IP address for that one

I assume in the map table?

@francislavoie

That’s probably why it doesn’t work – your backend Caddy can’t complete the TLS handshake because it’s expecting to see a domain in the handshake but you’re just giving it an IP address and it doesn’t have a certificate for that IP address.

I haven’t yet configured the backend. Shall I do that first and test before updating the map table with the domain name?

Time flies when you’re having fun. It’s almost 5 AM and I need to get some sleep. I’ll report back later on today.

[francislavoie]

@basil

Please correct me if I’m wrong, but I’m interpreting this as 'add a route wrapper for all the surrounding handle checks`.

No I mean literally change the word handle with route for any of the blocks that you want to actually fall through to another one after being matched. The handle directive is specifically designed to not run other handle blocks after the first matching one is run.

@basil

I assume in the map table?

Yeah.

@basil

I haven’t yet configured the backend.

Oh, well the proxy obviously won’t work then. It won’t be able to connect over TLS if the thing on the other end isn’t expecting that.

[basil (Basil Hendroff)]

So, before proceeding with backend configuration, I thought it useful to take stock and highlight key changes in my environment relating to mTLS configuration.

Local DNS additions

acme.lan → 10.1.1.4
test.lan → 10.1.1.50

Relevant frontend Caddyfile mTLS constructs

...
# ACME server
acme.lan {
  acme_server
  tls internal
}
...
*.udance.com.au {
...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
...
    test         test.lan:443    yes    yes  yes        # test.udance.com.au
...
# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  route @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

The next stop is backend configuration, however, before proceeding, I’d like to review my understanding of the route vs handle directives. A lot has happened in the last few posts that spun me out. A separate post to follow…

[basil (Basil Hendroff)]

@francislavoie

change the word handle with route for any of the blocks that you want to actually fall through to another one after being matched. The handle directive is specifically designed to not run other handle blocks after the first matching one is run.

This was one of those lightbulb moments for me. I had read @matt’s excellent wiki article Composing in the Caddyfile many times over and I’ll continue to read it over and over again I’m sure. There’s so much to take in. Sometimes though, saying things differently can make a world of difference. This was one of those times for me.

I’ve gone through my Caddyfile with a fine-tooth comb reviewing places where I used the handle directive. I’d like to check off where I’ve replaced this with the route directive. This will either confirm for me that I understand the differences in these two directives, or, that I still need to reshape and deepen my understanding.

(online) {
  @offline expression `"{args.0}" == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `"{args.0}" == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  route @split {
    redir https://udance.statuspage.io temporary
  }
}

In the above snippet, I used the handle directive with the @offline matcher because I figured that if there is a match here, I won’t need to run any other handle blocks after this one. On the other hand, I used the route directive with the @split matcher. If there’s a match, then for the external network, the redir is invoked. For the internal network, however, I might still want to fall through to other handle blocks outside this snippet if this snippet happens to be invoked. Is this logic sound?

I applied similar logic to other handle blocks within the wildcard subdomain Caddy block…

*.udance.com.au {
...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
...
    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling

  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  route @split {
    redir https://udance.statuspage.io temporary
  }

# Authenticate phpMyAdmin on production WordPress sites

  @phpmyadmin expression `{phpmyadmin} == "yes"`
  route @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.

  @nc-apache host nc-apache.udance.com.au
  route @nc-apache {
    redir /.well-known/carddav /remote.php/carddav 301
    redir /.well-known/caldav /remote.php/caldav 301
  }

# Enable HSTS for Nextcloud

  @hsts host cloud.udance.com.au
  route @hsts {
    header {
      Strict-Transport-Security max-age=31536000;
    }
  }

# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  route @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

# Unsecured backend communication

  @nomtls expression `{mtls} == "no"`
  route @nomtls {
    reverse_proxy {backend}
  }
}
...

[francislavoie]

@basil

In the above snippet, I used the handle directive with the @offline matcher because I figured that if there is a match here, I won’t need to run any other handle blocks after this one. On the other hand, I used the route directive with the @split matcher. If there’s a match, then for the external network, the redir is invoked. For the internal network, however, I might still want to fall through to other handle blocks outside this snippet if this snippet happens to be invoked. Is this logic sound?

Redirects are always terminal (as in nothing runs after that) because they straight up write a response right away. Same with file_server, respond, reverse_proxy, etc.

In that case, since redir is ordered high in the directive order, you can remove the route/handle wrapping them and put the matcher on redir itself. So like:

(online) {
  @offline expression `"{args.0}" == "no"`
  redir @offline https://udance.statuspage.io temporary

  @split {
    expression `"{args.0}" == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  redir @split https://udance.statuspage.io temporary
}

And you can also reduce the HSTS one to:

  @hsts host cloud.udance.com.au
  header @hsts Strict-Transport-Security max-age=31536000;

The only ones that you had that were problematic were basicauth (i.e. import authorise), header for HSTS (improved with the above), and the @nc-apache (because it only matches two specific paths and then should fall through for anything else, route is correct there). The rest can be handle.

[basil (Basil Hendroff)]

@francislavoie

Redirects are always terminal (as in nothing runs after that) because they straight up write a response right away. Same with file_server, respond, reverse_proxy, etc.

Ahh…so, in a sense, directives like redir, etc…that are terminal trump handle and route.