mTLS under FreeBSD

You can ignore that message entirely if you don’t use Firefox on that machine. That’s just the underlying smallstep libs trying to install the root cert for Firefox since it doesn’t use the system’s trust store.

3 Likes

@Whitestrake @francislavoie Phew! I was beginning to think that was the end of my mTLS journey :cold_sweat:

For a start, I had no idea where that would be. :face_with_hand_over_mouth:

I also spotted this thread that had me worried as it referred to FreeBSD Starting with caddy2 - basic Caddyfile trying to use port 80

Thank goodness!

2 Likes

As I want to be selective about which backend services should have secured comms, I altered the map table and added a couple of handles in the wildcard domain Caddy block to give me that flexibility.

*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
    ...
    # Jails
    ...
    test         10.1.1.50:80    yes    yes  yes        # test.udance.com.au

    ...
  }
  ...
# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    # Watch this space!
  }

# Unsecured backend communication

  @nomtls expression `{mtls} == "no"`
  handle @nomtls {
    reverse_proxy {backend}
  }
}

Jump in anytime if you think there is a better way of going about this.

I should have tested out the modified wildcard domain before continuing. It turns out it wasn’t working the way I expected it to. Attempting to access a subdomain (when {mtls} is either yes and no} returned a blank screen.

An extract from caddy adapt --pretty for the unmodified wildcard domain block…

                                                                                {
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "reverse_proxy",
                                                                                                        "upstreams": [
                                                                                                                {
                                                                                                                        "dial": "{backend}"
                                                                                                                }
                                                                                                        ]
                                                                                                }
                                                                                        ]
                                                                                }

…and from the modified wildcard domain block…

                                                                                {
                                                                                        "group": "group28",
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "subroute"
                                                                                                }
                                                                                        ],
                                                                                        "match": [
                                                                                                {
                                                                                                        "expression": "{mtls} == \"yes\""
                                                                                                }
                                                                                        ]
                                                                                },
                                                                                {
                                                                                        "group": "group28",
                                                                                        "handle": [
                                                                                                {
                                                                                                        "handler": "subroute",
                                                                                                        "routes": [
                                                                                                                {
                                                                                                                        "handle": [
                                                                                                                                {
                                                                                                                                        "handler": "reverse_proxy",
                                                                                                                                        "upstreams": [
                                                                                                                                                {
                                                                                                                                                        "dial": "{backend}"
                                                                                                                                                }
                                                                                                                                        ]
                                                                                                                                }
                                                                                                                        ]
                                                                                                                }
                                                                                                        ]
                                                                                                }
                                                                                        ],
                                                                                        "match": [
                                                                                                {
                                                                                                        "expression": "{mtls} == \"no\""
                                                                                                }
                                                                                        ]

It appears the handler changed from reverse_proxy to subroute with the modified wildcard subdomain. After reviewing @matt’s wiki article Composing in the Caddyfile and with a bit of experimentation, I found that wrapping the matcher and associated handle directive in a route block fixed the issue.

This was confirmed in this caddy adapt --pretty extract;

                          "handle": [
                                  {
                                          "handler": "subroute",
                                          "routes": [
                                                  {
                                                          "handle": [
                                                                  {
                                                                          "handler": "subroute"
                                                                  }
                                                          ],
                                                          "match": [
                                                                  {
                                                                          "expression": "{mtls} == \"yes\""
                                                                  }
                                                          ]
                                                  }
                                          ]
                                  },
                                  {
                                          "handler": "subroute",
                                          "routes": [
                                                  {
                                                          "handle": [
                                                                  {
                                                                          "handler": "subroute",
                                                                          "routes": [
                                                                                  {
                                                                                          "handle": [
                                                                                                  {
                                                                                                          "handler": "reverse_proxy",
                                                                                                          "upstreams": [
                                                                                                                  {
                                                                                                                          "dial": "{backend}"
                                                                                                                  }
                                                                                                          ]
                                                                                                  }
                                                                                          ]
                                                                                  }
                                                                          ]
                                                                  }
                                                          ],
                                                          "match": [
                                                                  {
                                                                          "expression": "{mtls} == \"no\""
                                                                  }
                                                          ]
                                                  }
                                          ]
                                  }
                          ]

This is the working wildcard subdomain Caddy block…

*.udance.com.au {
  ...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
    ...
    # Jails
    ...
    test         10.1.1.50:80    yes    yes  yes        # test.udance.com.au

    ...
  }
  ...
# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      # Watch this space!
    }
  }

# Unsecured backend communication

  route {
    @nomtls expression `{mtls} == "no"`
    handle @nomtls {
      reverse_proxy {backend}
    }
  }
}
1 Like

So [head scratching], what do I replace ‘# Watch this space!’ with?

From @Rob789’s inspirational wiki article Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts , a quote from the section Local (Split) DNS

I created domain names for the hosts because pointing directly to the IP addresses didn’t work with the ACME server. I was not able to get certificates for the backends.

So, to set the scene, my local DNS resolver (DNSMasq if anyone is interested) resolves:

  1. caddy.lan → 10.1.1.4
  2. test.lan → 10.1.1.50

…and this extract from the section Local HTTPS

Secondly, you need define or update the FQDN where Caddy listens to and reverse proxies accordingly with TLS.

nextcloud.my.domain.com {
         reverse_proxy https://office.roadrunner {
              header_up Host {http.reverse_proxy.upstream.hostport}
              header_up X-Forwarded-Host {host}
         }
}

If I wasn’t using the map directive, the equivalent Caddy block for me would be:

test.udance.com.au {
         reverse_proxy https://test.lan {
              header_up Host {http.reverse_proxy.upstream.hostport}
              header_up X-Forwarded-Host {host}
         }
}

Using the map directive, here’s my first attempt on paper:

# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
      }
    }
  }

I’m not sure that the translation is correct? For a start, {backend} refers to an IP address and port number rather than a local domain name as suggested in the wiki article. Thoughts?

1 Like

Almost. You need to enable HTTPS, and because of the other topic where I explained all that, you can’t use the scheme, so you have to use the “long way” of configuring the transport:

    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
        transport http {
          tls
        }
      }
    }

Also you don’t need the route wrapping, I don’t see what that gives you here. FYI both handle and route directives generate subroutes, it’s just that they have different effects on how the Caddyfile is parsed (route overrides the directive order – not useful here), and whether they become mutually exclusive or not (i.e. handle).

2 Likes

Hi Basil,

Good luck with your journey. I will try to follow it and help where I can.

Just one warning. My setup is not working 100%. There is something going wrong with the renewal of the certificates. They get renewed but I cannot establish a connection between the nodes anymore. I have to stop all Caddy services, delete the issued certificates in the upstream node and restart everything again.

I will most likely open a new post topic for this as soon as I get some time to look into this more.

2 Likes

Curious. Referring to post #7 above, without the route wrapper, a browser (both Google Chrome and Microsoft Edge tested) returns nothing back. To demonstrate what I’m seeing now:

I start with this…

    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse proxy {backend}
    }

From a browser, when I try to access test.udance.com.au, an empty screen is returned.

mtls1

If I reinstate the route wrapper…

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse proxy {backend}
    }
  }

From a browser, I can now access the site test.udance.com.au

mtls2

If I now add in the code to support mTLS…

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
        transport http {
          tls
        }
      }
    }
  }

From a browser, I get a 502 error returned. I’m not sure if this is an expected result at this stage as I haven’t added anything yet to the backend Caddyfile. Note: Removing the route wrapper, but leaving the additional code returns an empty screen again.

mtls3

Oh if your map still has :80, then it needs to be :443, cause HTTPS is over port 443.

Re route, there must be something else going on in your config, cause there’s no reason for that to be the case.

Also to make reading the caddy adapt --pretty output less annoying, you can enter tabs -2 to set the tab size to 2 spaces for your current shell session.

If I write a Caddyfile like this:

*.udance.com.au {
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
    # Jails
    test         10.1.1.50:80    yes    yes  yes        # test.udance.com.au
  }
# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    reverse_proxy {backend}
  }

# Unsecured backend communication

  @nomtls expression `{mtls} == "no"`
  handle @nomtls {
    reverse_proxy {backend}
  }
}

Then adapt it, I get this, which all makes sense (subroutes in the right place etc)

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "*.udance.com.au"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "destinations": [
                            "{backend}",
                            "{online}",
                            "{mtls}",
                            "{phpmyadmin}"
                          ],
                          "handler": "map",
                          "mappings": [
                            {
                              "input": "test",
                              "outputs": [
                                "10.1.1.50:80",
                                "yes",
                                "yes",
                                "yes"
                              ]
                            }
                          ],
                          "source": "{http.request.host.labels.3}"
                        }
                      ]
                    },
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "{backend}"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "expression": "{mtls} == \"yes\""
                        }
                      ]
                    },
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "reverse_proxy",
                                  "upstreams": [
                                    {
                                      "dial": "{backend}"
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "expression": "{mtls} == \"no\""
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

(Also you could pipe the output into jq, like caddy adapt | jq which will syntax highlight it and format it. See jq )

2 Likes

Changed that. Still getting the 502 error.

Thanks for the tip :grinning:

Also, good to know. :grinning:

I had hoped to only highlight the minimal parts of my Caddyfile that are relevant to solving the mTLS puzzle, but for the sake of completeness, here are the relevant parts of my Caddyfile, including the global section and snippets, which relate to the udance.com.au domain.

{
  email basil.hendroff@udance.com.au
  acme_dns cloudflare [REDACTED]
#  debug
  log {
    format json {
      time_format iso8601
    }
  }
}

#----------------------------------------------------------------------
# Snippet   : Description                           : Arguments
#----------------------------------------------------------------------
# authorise : Basic authentication                  : subdirectory
# logging   : Rolling access log                    : subdomain
# online    : Domain availability                   : {yes|no|split}

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

(logging) {
  log {
    format json {
      time_format iso8601
    }
    output file /var/log/caddy/{args.0}.log {
      roll_size 100MiB    # Default 100MiB
      roll_keep 10        # Default 10
      roll_keep_for 90d   # Default 90d
    }
  }
}

www.udance.com.au {
  redir https://udance.com.au{uri} permanent
}

udance.com.au {

  encode gzip zstd
  import logging udance.com.au
  import authorise /phpmyadmin*

  map {path} {backend} {online} {

#   PATH                BACKEND          ONLINE
#---------------------------------------------------------------
    ~^/tautulli.*       10.1.1.26:8181   yes
    ~^/transmission.*   10.1.1.28:9091   yes
    ~^/.*               10.1.1.55:80     yes
  }

# Offline handling

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  handle @split {
    redir https://udance.statuspage.io temporary
  }

  reverse_proxy {backend}
}


*.udance.com.au {

  encode gzip zstd
  import logging udance.com.au

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    # Docker containers

    office       10.1.1.13:8880  yes    no   no         # OnlyOffice
    portainer    10.1.1.13:9000  yes    no   no         # Portainer
    truecommand  10.1.1.13:8080  yes    no   no         # TrueCommand
    tc123        10.1.1.13:8082  yes    no   no         # TrueCommand v1.2.3
    nc-fpm       10.1.1.13:8031  yes    no   no         # Nextcloud+Caddy
    wordpress    10.1.1.13:5050  yes    no   no         # WordPress
    nc-apache    10.1.1.13:8030  yes    no   no         # Nextcloud+Apache
    collabora    10.1.1.13:9980  yes    no   no         # Collabora

    # Jails

    rslsync      10.1.1.22:8888  yes    no   no         # Resilio Sync
    cloud        10.1.1.29:80    yes    no   no         # Nextcloud
    heimdall     10.1.1.23:80    yes    no   no         # Heimdall
    blog         10.1.1.54:80    yes    no   yes        # blog.udance.com.au
    test         10.1.1.50:443   yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling

  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  handle @split {
    redir https://udance.statuspage.io temporary
  }

# Authenticate phpMyAdmin on production WordPress sites

  @phpmyadmin expression `{phpmyadmin} == "yes"`
  handle @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.

  @nc-apache host nc-apache.udance.com.au
  handle @nc-apache {
    redir /.well-known/carddav /remote.php/carddav 301
    redir /.well-known/caldav /remote.php/caldav 301
  }

# Enable HSTS for Nextcloud

  @hsts host cloud.udance.com.au
  handle @hsts {
    header {
      Strict-Transport-Security max-age=31536000;
    }
  }

# Secure backend communication

  route {
    @mtls expression `{mtls} == "yes"`
    handle @mtls {
      reverse_proxy {backend} {
        header_up Host {http.reverse_proxy.upstream.hostport}
        header_up X-Forwarded-Host {host}
        transport http {
          tls
        }
      }
    }
  }

# Unsecured backend communication

  route {
    @nomtls expression `{mtls} == "no"`
    handle @nomtls {
      reverse_proxy {backend}
    }
  }
}

Ah, it’s cause handle is mutually exclusive, so since you’re using that all over the place beforehand, it only ever runs the first matching handle block. Most of those should be changed to route instead probably since you do want them to fall through (like the ones for headers and basicauth). Specifically I think cause you set phpmyadmin to yes for that domain, it hits that handle then never runs the one with mtls.

Also I think you probably want to use the actual domain instead of the IP address for that one (and its domain needs to resolve to that IP address with your internal DNS server). That’s probably why it doesn’t work – your backend Caddy can’t complete the TLS handshake because it’s expecting to see a domain in the handshake but you’re just giving it an IP address and it doesn’t have a certificate for that IP address. I think. But check your logs.

Please correct me if I’m wrong, but I’m interpreting this as 'add a route wrapper for all the surrounding handle checks`.

Oh, I see.

I assume in the map table?

I haven’t yet configured the backend. Shall I do that first and test before updating the map table with the domain name?

Time flies when you’re having fun. It’s almost 5 AM and I need to get some sleep. I’ll report back later on today.

No I mean literally change the word handle with route for any of the blocks that you want to actually fall through to another one after being matched. The handle directive is specifically designed to not run other handle blocks after the first matching one is run.

Yeah.

Oh, well the proxy obviously won’t work then. It won’t be able to connect over TLS if the thing on the other end isn’t expecting that.

2 Likes

So, before proceeding with backend configuration, I thought it useful to take stock and highlight key changes in my environment relating to mTLS configuration.

Local DNS additions

acme.lan → 10.1.1.4
test.lan → 10.1.1.50

Relevant frontend Caddyfile mTLS constructs

...
# ACME server
acme.lan {
  acme_server
  tls internal
}
...
*.udance.com.au {
...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
...
    test         test.lan:443    yes    yes  yes        # test.udance.com.au
...
# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  route @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

The next stop is backend configuration, however, before proceeding, I’d like to review my understanding of the route vs handle directives. A lot has happened in the last few posts that spun me out. A separate post to follow…

:bulb:This was one of those lightbulb moments for me. I had read @matt’s excellent wiki article Composing in the Caddyfile many times over and I’ll continue to read it over and over again I’m sure. There’s so much to take in. Sometimes though, saying things differently can make a world of difference. This was one of those times for me.

I’ve gone through my Caddyfile with a fine-tooth comb reviewing places where I used the handle directive. I’d like to check off where I’ve replaced this with the route directive. This will either confirm for me that I understand the differences in these two directives, or, that I still need to reshape and deepen my understanding.

(online) {
  @offline expression `"{args.0}" == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `"{args.0}" == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  route @split {
    redir https://udance.statuspage.io temporary
  }
}

In the above snippet, I used the handle directive with the @offline matcher because I figured that if there is a match here, I won’t need to run any other handle blocks after this one. On the other hand, I used the route directive with the @split matcher. If there’s a match, then for the external network, the redir is invoked. For the internal network, however, I might still want to fall through to other handle blocks outside this snippet if this snippet happens to be invoked. Is this logic sound?

I applied similar logic to other handle blocks within the wildcard subdomain Caddy block…

*.udance.com.au {
...
  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------
...
    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling

  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline

  @offline expression `{online} == "no"`
  handle @offline {
    redir https://udance.statuspage.io temporary
  }

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  route @split {
    redir https://udance.statuspage.io temporary
  }

# Authenticate phpMyAdmin on production WordPress sites

  @phpmyadmin expression `{phpmyadmin} == "yes"`
  route @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.

  @nc-apache host nc-apache.udance.com.au
  route @nc-apache {
    redir /.well-known/carddav /remote.php/carddav 301
    redir /.well-known/caldav /remote.php/caldav 301
  }

# Enable HSTS for Nextcloud

  @hsts host cloud.udance.com.au
  route @hsts {
    header {
      Strict-Transport-Security max-age=31536000;
    }
  }

# Secure backend communication

  @mtls expression `{mtls} == "yes"`
  route @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

# Unsecured backend communication

  @nomtls expression `{mtls} == "no"`
  route @nomtls {
    reverse_proxy {backend}
  }
}
...
1 Like

Redirects are always terminal (as in nothing runs after that) because they straight up write a response right away. Same with file_server, respond, reverse_proxy, etc.

In that case, since redir is ordered high in the directive order, you can remove the route/handle wrapping them and put the matcher on redir itself. So like:

(online) {
  @offline expression `"{args.0}" == "no"`
  redir @offline https://udance.statuspage.io temporary

  @split {
    expression `"{args.0}" == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  redir @split https://udance.statuspage.io temporary
}

And you can also reduce the HSTS one to:

  @hsts host cloud.udance.com.au
  header @hsts Strict-Transport-Security max-age=31536000;

The only ones that you had that were problematic were basicauth (i.e. import authorise), header for HSTS (improved with the above), and the @nc-apache (because it only matches two specific paths and then should fall through for anything else, route is correct there). The rest can be handle.

1 Like

Ahh…so, in a sense, directives like redir, etc…that are terminal trump handle and route.

Uh, I wouldn’t really word it that way, but sure :stuck_out_tongue:

Anyways if what I said was unclear, from your current config I’d write it like this:

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

*.udance.com.au {

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

# Error handling
  @unknown expression `{backend} == "unknown"`
  handle @unknown {
    respond "Denied" 403
  }

# Site offline
  @offline expression `{online} == "no"`
  redir @offline https://udance.statuspage.io temporary

  @split {
    expression `{online} == "split"`
    not remote_ip 10.1.1.0/24 10.1.2.0/24
  }
  redir @split https://udance.statuspage.io temporary

# Authenticate phpMyAdmin on production WordPress sites
  @phpmyadmin expression `{phpmyadmin} == "yes"`
  route @phpmyadmin {
    import authorise /phpmyadmin*
  }

# Fix when using the Nextcloud+Apache Docker image with Caddy.
  @nc-apache host nc-apache.udance.com.au
  route @nc-apache {
    redir /.well-known/carddav /remote.php/carddav permanent
    redir /.well-known/caldav /remote.php/caldav permanent
  }

# Enable HSTS for Nextcloud
  @hsts host cloud.udance.com.au
  header @hsts "Strict-Transport-Security max-age=31536000;"

# Secure backend communication
  @mtls expression `{mtls} == "yes"`
  handle @mtls {
    reverse_proxy {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }
  }

# Unsecured backend communication
  @nomtls expression `{mtls} == "no"`
  handle @nomtls {
    reverse_proxy {backend}
  }
}

No, you were very clear, but several related questions have sprung up…

As reverse_proxy was used here…

…could I have changed this to…

# Unsecured backend communication
  @nomtls expression `{mtls} == "no"`
  reverse_proxy @nomtls {backend}

…and as respond was used here…

…could this have been reduced as well to…

# Error handling
  @unknown expression `{backend} == "unknown"`
  respond @unknown "Denied" 403

Also, if I move the handle block below further up above other handle blocks, or, happen to add a new handle block after this one, would I need to change the handle directive to route so that if there’s a match, it runs through to the next handle block?

…or, because I’m using reverse_proxy, which is terminal, to avoid any ambiguity, would this work?

# Secure backend communication
  @mtls expression `{mtls} == "yes"`
  reverse_proxy @mtls {backend} {
    header_up Host {http.reverse_proxy.upstream.hostport}
    header_up X-Forwarded-Host {host}
    transport http {
      tls
    }
  }

If you unwrap those, then you lose the benefits of handle's mutual exclusivity and directive sorting order (which is guaranteed to be in the order they are written in the Caddyfile in relation to eachother) and the directives like respond and reverse_proxy would get sorted according to their directive order. The effect of this would be that respond @unknown "Denied" 403 would happen after the redir and header and basicauth which is probably not correct (doesn’t make sense to trigger auth when you’re going to respond with a 403 for example).

That said, I realized I made a mistake, because route is ordered after handle, so all the route blocks will get put at the end :man_facepalming: (you can see this in the adapt output).

The dumb/easy fix it to just wrap the whole thing in a single top-level route to tell the Caddyfile adapter “no, do it in this order” but that’s not so nice :grimacing: but in that case you don’t need some of the handle anymore because you can safely place directives in your Caddyfile in the right order by hand.

Ultimately, you’re writing a super complex config here, so it’s pretty tricky to get right (cause you have multiple paths through which a single request could go because you have multiple variables at play).

So this should work I guess:

(authorise) {
  basicauth {args.0} {
    admin [REDACTED]
  }
}

*.udance.com.au {

  map {labels.3} {backend} {online} {mtls} {phpmyadmin} {

#   HOSTNAME     BACKEND         ONLINE mTLS PHPMYADMIN #COMMENT
#---------------------------------------------------------------

    test         test.lan:443    yes    yes  yes        # test.udance.com.au
    basil        10.1.1.56:80    yes    no   yes        # basil.udance.com.au
    sachika      10.1.1.57:80    yes    no   yes        # sachika.udance.com.au
    default      unknown         yes    no   no         # subdomain does not exist
  }

  route {
# Error handling
    @unknown expression `{backend} == "unknown"`
    respond @unknown "Denied" 403

# Site offline
    @offline expression `{online} == "no"`
    redir @offline https://udance.statuspage.io temporary

    @split {
      expression `{online} == "split"`
      not remote_ip 10.1.1.0/24 10.1.2.0/24
    }
    redir @split https://udance.statuspage.io temporary

# Authenticate phpMyAdmin on production WordPress sites
    @phpmyadmin expression `{phpmyadmin} == "yes"`
    route @phpmyadmin {
      import authorise /phpmyadmin*
    }

# Fix when using the Nextcloud+Apache Docker image with Caddy.
    @nc-apache host nc-apache.udance.com.au
    route @nc-apache {
      redir /.well-known/carddav /remote.php/carddav permanent
      redir /.well-known/caldav /remote.php/caldav permanent
    }

# Enable HSTS for Nextcloud
    @hsts host cloud.udance.com.au
    header @hsts "Strict-Transport-Security max-age=31536000;"

# Secure backend communication
    @mtls expression `{mtls} == "yes"`
    reverse_proxy @mtls {backend} {
      header_up Host {http.reverse_proxy.upstream.hostport}
      header_up X-Forwarded-Host {host}
      transport http {
        tls
      }
    }

# Unsecured backend communication
    @nomtls expression `{mtls} == "no"`
    reverse_proxy @nomtls {backend}

  }
}
2 Likes