V2: understanding rewrite

lexpierce · April 29, 2020, 6:01am

Using 2.0.0 rc3. I have a file in /var/www/mta-sts/mta-sts.txt, and I want to serve it as https://mta-sts.meeple.ninja/.well-known/mta-sts.txt

Basic stuff.

Here is the handlers for that domain:

            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "Strict-Transport-Security": [
                                "max-age=63072000; includeSubDomains; preload"
                              ]
                            }
                          }
                        },
                        {
                          "encodings": {
                            "gzip": {},
                            "zstd": {}
                          },
                          "handler": "encode"
                        },
                        {
                          "handler": "file_server",
                          "root": "/var/www/mta-sts"
                        },
                        {
                          "handler": "rewrite",
                          "strip_path_prefix": "/.well-known"
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/*"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "mta-sts.meeple.ninja"
                  ]
                }
              ],
              "terminal": true
            }

Here is the log output…

{
  "level": "error",
  "ts": 1588139759.3504975,
  "logger": "http.log.access.mta-sts",
  "msg": "handled request",
  "request": {
    "method": "GET",
    "uri": "/.well-known/mta-sts.txt",
    "proto": "HTTP/2.0",
    "remote_addr": "184.155.141.233:63863",
    "host": "mta-sts.meeple.ninja",
    "headers": {
      "Cache-Control": [
        "max-age=0"
      ],
      "Te": [
        "trailers"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0"
      ],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
      ],
      "Accept-Language": [
        "en-US,en;q=0.5"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ],
      "Dnt": [
        "1"
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "ciphersuite": 4865,
      "proto": "h2",
      "proto_mutual": true,
      "server_name": "mta-sts.meeple.ninja"
    }
  },
  "common_log": "184.155.141.233 - - [28/Apr/2020:22:55:59 -0700] \"GET /.well-known/mta-sts.txt HTTP/2.0\" 404 0",
  "latency": 0.000132348,
  "size": 0,
  "status": 404,
  "resp_headers": {
    "Server": [
      "Caddy"
    ],
    "Strict-Transport-Security": [
      "max-age=63072000; includeSubDomains; preload"
    ]
  }
}

Is my issue here in the matcher? Am I not quite understanding re-write rules? This is something that was dead simple in v1, and in NGINX, and it is really stumping me here.

Note: Doing this all in JSON because the log writing rules in Caddyfile suck, and the JSON output by adapt is kinda wack.

lexpierce · April 29, 2020, 6:14am

Used adapt and realized I maybe needed the rewrite as a peer above the matcher for “/”, so did this change. Same result:

            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "Strict-Transport-Security": [
                                "max-age=63072000; includeSubDomains; preload"
                              ]
                            }
                          }
                        },
                        {
                          "encodings": {
                            "gzip": {},
                            "zstd": {}
                          },
                          "handler": "encode"
                        },
                        {
                          "handler": "file_server",
                          "root": "/var/www/mta-sts"
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "rewrite",
                          "strip_path_prefix": "/.well-known"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "mta-sts.meeple.ninja"
                  ]
                }
              ],
              "terminal": true
            }

and the log:

{
  "level": "info",
  "ts": 1588140562.4342704,
  "logger": "http.log.access.mta-sts",
  "msg": "handled request",
  "request": {
    "method": "GET",
    "uri": "/.well-known/mta-sts.txt",
    "proto": "HTTP/2.0",
    "remote_addr": "184.155.141.233:63985",
    "host": "mta-sts.meeple.ninja",
    "headers": {
      "Dnt": [
        "1"
      ],
      "Upgrade-Insecure-Requests": [
        "1"
      ],
      "Cache-Control": [
        "max-age=0"
      ],
      "Te": [
        "trailers"
      ],
      "User-Agent": [
        "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0"
      ],
      "Accept": [
        "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
      ],
      "Accept-Language": [
        "en-US,en;q=0.5"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br"
      ]
    },
    "tls": {
      "resumed": true,
      "version": 772,
      "ciphersuite": 4865,
      "proto": "h2",
      "proto_mutual": true,
      "server_name": "mta-sts.meeple.ninja"
    }
  },
  "common_log": "184.155.141.233 - - [28/Apr/2020:23:09:22 -0700] \"GET /.well-known/mta-sts.txt HTTP/2.0\" 0 0",
  "latency": 9.8846e-05,
  "size": 0,
  "status": 0,
  "resp_headers": {
    "Server": [
      "Caddy"
    ]
  }
}

this time I get a status ‘0’ instead of ‘404’

btw, here is the adapt output… trying not to use separate vars section, passing root directly to file_server:

            {
              "match": [
                {
                  "host": [
                    "mta-sts.meeple.ninja"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "vars",
                          "root": "/var/www/mta-sts"
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "Strict-Transport-Security": [
                                "max-age=63072000; includeSubDomains; preload"
                              ]
                            }
                          }
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "rewrite",
                          "strip_path_prefix": "/.well-known"
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }

lexpierce · April 29, 2020, 6:19am

Aha ha! So rewrite rules and file_server need to be outside the matcher for “/”. Hmm… Did I miss anything about modules and nesting/peer level in the docs?

Here is the working solution

            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "headers",
                          "response": {
                            "set": {
                              "Strict-Transport-Security": [
                                "max-age=63072000; includeSubDomains; preload"
                              ]
                            }
                          }
                        },
                        {
                          "encodings": {
                            "gzip": {},
                            "zstd": {}
                          },
                          "handler": "encode"
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "rewrite",
                          "strip_path_prefix": "/.well-known"
                        },
                        {
                          "handler": "file_server",
                          "root": "/var/www/mta-sts"
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "mta-sts.meeple.ninja"
                  ]
                }
              ],
              "terminal": true
            }

lexpierce · April 29, 2020, 6:21am

I think I am starting to see why the nextcloud way of using multiple root XXX.php file (remote|index|etc).php is painful with this new config. This is going to make transition some sites over to v2 difficult.

Whitestrake · April 29, 2020, 6:41am

You say you want to serve the file /var/www/mta-sts/mta-sts.txt from a request to /.well-known/mta-sts.txt, but none of your examples reference /.well-known/mta-sts.txt at all. Your first one path matches to /* (pretty much literally any path - you can just remove the matcher entirely at this point), and a later one path matches / (literally ONLY a request to webroot, so /foo and /.well-known/mta-sts.txt get nothing).

What behaviour do you actually want to see?

francislavoie · April 29, 2020, 6:45am

FWIW the major outstanding issues with NextCloud regarding path handling should mostly be fixed by this PR which will be in v2.0 stable:

github.com/caddyserver/caddy

matcher: Add `split_path` option to file matcher; used in php_fastcgi

caddyserver:master ← francislavoie:try-files-split-path

opened 01:00AM - 23 Apr 20 UTC

francislavoie

+342 -188

Discussion: https://caddy.community/t/help-to-migrate-caddyfile-v1-to-v2-for-nex…tcloud/7647/18 A common pattern in PHP apps is to have a PHP file that acts as a router entrypoint to the application. This is typically the `index.php` file at the webroot or some other subpath of the application. These entrypoints typically accept requests that have a path appended to the filename, for example `/index.php/blog/123`. Some PHP apps happen to have more than one router entrypoint, for example, `/router.php/some/path`, in addition to the `index.php` file. NextCloud is one of those applications; it has `remote.php`, `public.php` and more that act as entrypoints. With the default `php_fastcgi` Caddyfile directive, many of those pages didn't work correctly in NextCloud. The problem has to do with the `try_files` matcher and rewrite that is included as part of the `php_fastcgi` shortcut: ``` try_files {path} {path}/index.php index.php ``` The issue is that on a request to the path `/remote.php/dav/files`, Caddy would check if that exact path exists as a file on disk (i.e. the first argument, `{path}`), but this doesn't work because there's extra stuff after the actual filename. We already have an option in the reverse_proxy's fastcgi transport to allow splitting the path into two parts, with the left-hand side being the filename (including the split value), and the right-hand side being the path, i.e. `split .php`. The fix we landed on for this issue is to add a new `split` option to the file matcher (`split_path` in JSON) that allows for splitting the path the same way, before testing if the file exists on disk. It looks like this: ``` @phpFiles { file { split .php try_files {path} {path}/index.php index.php } } ``` So now for a request to `/remote.php/dav/files`, Caddy will instead look for `/remote.php` on disk while matching. The `php_fastcgi` shortcut is also updated to internally add this new split option for its `try_files` matcher. This should not be a breaking change, because we assume that any path that involved `.php` in the middle of it was assumed to be the filename boundary, but this wasn't working as expected.

lexpierce · April 29, 2020, 6:47am

See later configs. I realize the star matcher was out. I guess I could set a matcher so that only the mta-sts.txt file is served. the main thing here is I would like to serve it from /var/www/mta-sts, and not /var/www/mta-sts/.well-known.

I think I have that part figured out. Just understanding that:

Rewrite of URI needs to take place outside of “/” matcher.
file_server and filesystem root definition needs to take place outside of “/” matcher

are what I wasn’t quite understanding. I think I have that down now.

lexpierce · April 29, 2020, 6:49am

That’s excellent! This will be nice, and mean that the php_fastcgi directive will be easier to use. I will then of course adapt it and figure out how that makes a difference in the JSON.

Whitestrake · April 29, 2020, 6:50am

I mean, I still don’t understand exactly what you’re after.

Requests to / exactly (web root) get served /var/www/mta-sts/mta-sts.txt ??
Requests to /.well-known/mta-sts.txt get served the same file ??
Requests to anywhere else get… Nothing ?

If you can clarify the intended behaviour in plain English, I might be able to tell you what kind of config will achieve it.

lexpierce · April 29, 2020, 6:52am

Requests to /.well-known/mta-sts.txt get served /var/www/mta-sts/mta-sts.txt. Nothing else is served.

If this can be done simpler than what I am doing now that’d be great.

francislavoie · April 29, 2020, 6:54am

It’s as simple as this, ultimately:

root * /var/www/mta-sts
rewrite /.well-known/mta-sts.txt /mta-sts.txt
file_server

(You can adapt that to JSON)

lexpierce · April 29, 2020, 6:58am

Will this server anything in /var/www/mta-sts if it is known (browse is offm obviously)? Not that there are any other files there, more for curiosity sake on the scope of file_server

francislavoie · April 29, 2020, 6:58am

Yes. You can use a not matcher to make anything else respond with a 404 if you like.

Whitestrake · April 29, 2020, 6:59am

Much simpler than what you have.

Start with this Caddyfile:

mta-sts.meeple.ninja {
  handle /.well-known/mta-sts.txt {
    rewrite mta-sts.txt
    file_server {
      root /var/www/mta-sts
    }
  }
}

Adapt it:

{"apps":{"http":{"servers":{"srv0":{"listen":[":443"],"routes":[{"match":[{"host":["mta-sts.meeple.ninja"]}],"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"subroute","routes":[{"group":"group0","handle":[{"handler":"rewrite","uri":"mta-sts.txt"}]},{"handle":[{"handler":"file_server","hide":["Caddyfile"],"root":"/var/www/mta-sts"}]}]}],"match":[{"path":["/.well-known/mta-sts.txt"]}]}]}],"terminal":true}]}}}}}

Prettify, and do some light clean up:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "mta-sts.meeple.ninja"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "handler": "rewrite",
                                  "uri": "mta-sts.txt"
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "file_server",
                                  "root": "/var/www/mta-sts"
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/.well-known/mta-sts.txt"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

Result: Requests for any path get 404’d, except for /.well-known/mta-sts.txt, which serves /var/www/mta-sts/mta-sts.txt.

Rewrites are conceptually quite simple in a void of context, it’s how they interact that trips some people up. Just remember: All they do is modify the URI, which is effectively the client’s requested resource. Keep in mind that to a HTTP file server, the ROOT plus the PATH equals the file-on-disk to serve. So you have your ROOT /var/www/mta-sts, and you know which file you want to serve (mta-sts.txt relative to the ROOT), you need to rewrite so that the PATH looks how it should in order to add up to the file path /var/www/mta-sts/mta-sts.txt, then sicc the file server on it to handle the rest.

My approach has the benefit over @francislavoie that you are literally always rewriting to the exact file, and setting the web root, so only one file will ever be served - /var/www/mta-sts/mta-sts.txt. And only ever for one request - exactly /.well-known/mta-sts.txt, becuase that’s what handle is set to use. Everything else - no configuration, hence no handlers, hence 404s.

francislavoie · April 29, 2020, 7:03am

Actually not true, you’ll get empty 200s in v2 by default unless you configure it to 404.

lexpierce · April 29, 2020, 7:04am

Ok. I like this. so conceptualy, the matcher is only matching on /.well-known/mta-sts.txt, and then that match has a handle clause that has two handlers: rewrite to reduce the uri to just the filename, and file_server to serve the file from root location.

Got it. I am starting to see how this works, but sounds like I just need to reduce what I am doing to it’s basics. Thanks, this was a super useful answer!

Whitestrake · April 29, 2020, 7:04am

Bah, hoisted by my own petard!

A minor technicality

Whitestrake · April 29, 2020, 7:06am

Aye. In v1 even, we had loads of people overthinking their Caddyfiles. A major benefit of the Caddyfile is that you can do a lot of things very simply - the Caddyfile is simple, don’t overthink it!

Now we have JSON in v2, and it’s incredibly powerful - but orders of magnitude easier to overthink. Keep it as conceptually simple as possible!

lexpierce · April 29, 2020, 7:07am

I’ll configure it to 404. Should that be done with a not matcher?

say something like

respond not /.well-known/mta-sts.txt "Not found" 404 {
  close
}

francislavoie · April 29, 2020, 7:09am

@notWellKnown {
    not path /.well-known/mta-sts.txt
}
respond @notWellKnown "Not Found" 404

This will turn into a two-liner in v2.1, there’s a PR to make one-line named matchers possible.

The only thing you can put as the first param for directives when they accept matchers, is either a path (starts with /), a * for “anything”, or a named matcher (starts with @). The * is only necessary if your next param starts with a / because otherwise it’s ambiguous which param is which. This typically comes up with the root directive which has a path as the first argument, and absolute paths are usually the norm.

All explained here:

Edit: If you go with @Whitestrake’s approach instead using handle, then all you’d need to do is add respond "Not Found" 404 at the top-level of your site and anything that doesn’t match the handle will fall through to respond, without a need for a not matcher. It’s just flipping booleans. Depends how you want to think about it.