Matches, Handlers, Routes, oh my!

1. Caddy version (caddy version):

v2.2.1 h1:Q62GWHMtztnvyRU+KPOpw6fNfeCD3SkwH7SfT1Tgt2c=

2. How I run Caddy:

systemd on Ubuntu Linux in production; though right now I’m running caddy run on a Mac laptop for testing.

a. System environment:

Ubuntu Linux, eventually; OSX right now.

b. Command:

./caddy run

d. My complete Caddyfile or JSON config:

127.0.0.1:80 {
  @missingFile not file {path} {path}.md
  handle @missingFile {
    respond @missingFile "Not found" 404
  }

  @notes {
    path_regexp \/\d{4}\/?.*$
  }

  handle @notes {
    templates
    rewrite * /note.html
  }

  root * /Users/smerrill/skippy/content/
  templates
  file_server

  rewrite /* /index.html

}

3. The problem I’m having:

I don’t completely understand how named matches, routes, and handlers interact.

What I want is to serve different templates for different paths. Request for /foo should use the index.html template, and requests for /2020/10/30/foo should use the note.html template.

The old v1 Markdown system was simple and worked extremely well for my needs. I’m finding it hard to implement all the logic in my Caddyfile to make this work for v2.

4. Error messages and/or full log output:

2020/11/01 00:35:33.972	ERROR	http.log.error	template: /index.html:8:20: executing "/index.html" at <include $markdownFilePath>: error calling include: open /Users/smerrill/skippy/content/2020/10/30/foo.md: no such file or directory	{"request": {"remote_addr": "127.0.0.1:64903", "proto": "HTTP/1.1", "method": "GET", "host": "127.0.0.1", "uri": "/2020/10/30/foo", "headers": {"Connection": ["keep-alive"], "Upgrade-Insecure-Requests": ["1"], "Dnt": ["1"], "Sec-Gpc": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Encoding": ["gzip, deflate"], "Accept-Language": ["en-US,en;q=0.5"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0"]}}, "duration": 0.00057551, "status": 500, "err_id": "9wxq8khf6", "err_trace": "templates.(*Templates).executeTemplate (templates.go:315)"}
2020/11/01 00:35:34.019	ERROR	http.log.error	template: /index.html:8:20: executing "/index.html" at <include $markdownFilePath>: error calling include: open /Users/smerrill/skippy/content/favicon.ico.md: no such file or directory	{"request": {"remote_addr": "127.0.0.1:64904", "proto": "HTTP/1.1", "method": "GET", "host": "127.0.0.1", "uri": "/favicon.ico", "headers": {"User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0"], "Dnt": ["1"], "Sec-Gpc": ["1"], "Accept": ["image/webp,*/*"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate"], "Connection": ["keep-alive"], "Referer": ["http://127.0.0.1/2020/10/30/foo"]}}, "duration": 0.000710707, "status": 500, "err_id": "c4z2e8yzb", "err_trace": "templates.(*Templates).executeTemplate (templates.go:315)"}

5. What I already tried:

As you can see from the Caddyfile, I’m trying to set up different templates for different paths, and also trying to properly handle 404 errors.

I have two major paths in my site: /YYYY/MM/DD/XYZ for short-form notes, and then long-form posts off the root. I would like different templates for each of these.

When I visit http://127.0.01/2020/10/30/foo, I expect to get a 404 error because there is no such file. Instead, Caddy is trying to use the /index.html template. Even if the 404 fails, I would expect the /note.html template to be used.

Here is my index.html template, which is a basic effort to output a Markdown file:

<html>
<body>
{{$pathParts := splitList "/" .OriginalReq.URL.Path}}
<br />
{{$markdownFilename := default "index" (slice $pathParts 1 | join "/")}}
{{ $markdownFilename }}
{{$markdownFilePath := printf "/%s.md" $markdownFilename}}
{{$markdownFile := (include $markdownFilePath | splitFrontMatter)}}
{{$title := default $markdownFilename $markdownFile.Meta.title}}
{{ $title }}
</body>
</html>

And here is note.html which is intended to be used for the request I’m trying, and also a minimal debug effort:

<html>
<head><title>note</title></head>
<body>
{{$pathParts := splitList "/" .OriginalReq.URL.Path}}
<br />
{{$markdownFilename := default "index" (slice $pathParts 1 | join "/")}}
{{ $markdownFilename }}
{{$markdownFilePath := printf "/%s.md" $markdownFilename}}
{{$markdownFile := (include $markdownFilePath | splitFrontMatter)}}
{{$title := default $markdownFilename $markdownFile.Meta.title}}
{{ $title }}
</body>
</html>

The error message I posted above indicates that index.html is being used for the request.

6. Links to relevant resources:

V2 Markdown, Templates, and HTTP error codes - my original post in my effort to move from Caddy v1 to v2.

I think the key part of Caddy’s behaviour you missed is that directives are sorted according to a predetermined order. See the list here:

So in situations like this, the quickest way to see how Caddy behaves is to look at the underlying JSON config. You can do this by running caddy adapt --pretty --config /path/to/Caddyfile. If I adapt yours, it looks like this:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":80"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "127.0.0.1"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "vars",
                          "root": "/Users/smerrill/skippy/content/"
                        }
                      ]
                    },
                    {
                      "group": "group3",
                      "handle": [
                        {
                          "handler": "rewrite",
                          "uri": "/index.html"
                        }
                      ],
                      "match": [
                        {
                          "path": [
                            "/*"
                          ]
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "templates"
                        }
                      ]
                    },
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "handle": [
                                {
                                  "body": "Not found",
                                  "handler": "static_response",
                                  "status_code": 404
                                }
                              ],
                              "match": [
                                {
                                  "not": [
                                    {
                                      "file": {
                                        "try_files": [
                                          "{http.request.uri.path}",
                                          "{http.request.uri.path}.md"
                                        ]
                                      }
                                    }
                                  ]
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "not": [
                            {
                              "file": {
                                "try_files": [
                                  "{http.request.uri.path}",
                                  "{http.request.uri.path}.md"
                                ]
                              }
                            }
                          ]
                        }
                      ]
                    },
                    {
                      "group": "group2",
                      "handle": [
                        {
                          "handler": "subroute",
                          "routes": [
                            {
                              "group": "group1",
                              "handle": [
                                {
                                  "handler": "rewrite",
                                  "uri": "/note.html"
                                }
                              ]
                            },
                            {
                              "handle": [
                                {
                                  "handler": "templates"
                                }
                              ]
                            }
                          ]
                        }
                      ],
                      "match": [
                        {
                          "path_regexp": {
                            "pattern": "\\/\\d{4}\\/?.*$"
                          }
                        }
                      ]
                    },
                    {
                      "handle": [
                        {
                          "handler": "file_server",
                          "hide": [
                            "Caddyfile-t"
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

There’s plenty to talk about here. So in JSON config, the HTTP routes are executed in order, from top to bottom, first checking the matcher to decide whether it should run the handler.

You’ll notice that your rewrite ends up being the 2nd handler (after the root directive), so all your requests get rewritten to index.html.

What you actually want is to only rewrite if your other handle blocks didn’t run, so the solution is to wrap your rewrite in a handle with no matcher.

You’ll notice in the JSON, a group property. This essentially marks a handler as being mutually exclusive from others with the same group ID. The handle Caddyfile directive uses that concept, pairing it with a subroute, so handle blocks are mutually exclusive from eachother, i.e. if one is matched and handled, then others with the same group ID won’t be executed.

On top of this, the Caddyfile adapter does sorting based on the matchers a directive has. Basically it’s like this:

  • If it has a path matcher, it’s sorted higher by the length of the paths (so a longer path will be sorted first, as it’s probably more specific)
  • If it has no path matcher, but has any other kind of matcher, it will come next
  • If it has no matcher at all, then it comes last (fall-back, catch-all)

Putting all that together, this is what I think your config should likely be:

:80 {
	root * /Users/smerrill/skippy/content/

	@missingFile not file {path} {path}.md
	handle @missingFile {
		respond * "Not found" 404
	}

	@notes path_regexp \/\d{4}\/?.*$
	handle @notes {
		rewrite * /note.html
	}

	handle {
		rewrite * /index.html
	}

	templates
	file_server
}

I ordered the directives in the order that they will end up in the JSON, so it’s easier to understand how it works. This is the conventional way to write Caddyfiles.

A few other notes:

  • I changed /* to just * because it makes a difference for sorting, because /* is a path matcher, but * is just a “catch-all”, and it’s more efficient because it avoids a string comparison from the path matcher.
  • I removed the duplicated templates directive, because it doesn’t make sense to pass your content through it twice.
  • You can use the single-line matcher syntax for the @notes matcher
  • It’s redundant to use @missingFile inside of the handle, because you’ve already matched that. So you can just do * instead.

I hope this gets you further!

1 Like

Thank you! Your thorough explanation really helps me understand this better.

The modified Caddyfile you provided works as I described I wanted:

  • non-existent files get a 404
  • /2020/11/01/foo gets the note.html template
  • everything else gets the index.html template

I’ve noticed two things with this:

  1. accessing a directory (just /2020/11/01/) gets a 404. This is because I don’t have a browse setting in file_server, I believe.
  2. The default handler’s rewrite will pass everything through my template. I realize I was unclear in my original posting, as I only want to template Markdown files, and all other files should be served as is, if they exist. Think GIFs and JPGs: I don’t want to pass those through a template.

I believe I can build upon the excellent start you’ve given me to fix up these two issues. Thanks again!

1 Like

For that I think you’ll want to use the try_files directive:

Note that this one doesn’t support matchers, but you can put it inside handle which does have a matcher. This is because it itself is a shortcut for a file matcher + rewrite handler pair.

I’ll mark this as solved, but feel free to come back to ask for more clarification once you’ve played around some more! :smile:

Note that this one doesn’t support matchers, but you can put it inside handle which does have a matcher. This is because it itself is a shortcut for a file matcher + rewrite handler pair.

What I’d like is to gracefully handle each of the following URIs:

  • /2020/11/01/foo
  • /2020/11/01/
  • /2020/11/01

This is what I tried:

     @notes path_regexp \/\d{4}\/?.*$
     handle @notes {
         try_files {path} {path}.md {path}index.md {path}/index.md
         rewrite * /templates/note.html
     }

I have the following files on disk:

❯ pwd
/Users/smerrill/skippy/public/2020/05
❯ tree
.
├── 10
│   ├── 0908.md
│   └── 1143.md
├── index.md
└── index.txt

1 directory, 4 files

If I load 127.0.0.1/2020/05/10/1143 I get the template-rendered output I expect.

If I load 127.0.0.1/2020/05/10/ I get an error:

2020/11/01 14:54:22.410	ERROR	http.log.error	template: /templates/note.html:4:20: executing "/templates/note.html" at <include $markdownFilePath>: error calling include: open /Users/smerrill/skippy/public/2020/05/10/.md: no such file or directory	{"request": {"remote_addr": "127.0.0.1:55261", "proto": "HTTP/1.1", "method": "GET", "host": "127.0.0.1", "uri": "/2020/05/10/", "headers": {"User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0"], "Dnt": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate"], "Connection": ["keep-alive"], "Upgrade-Insecure-Requests": ["1"], "Sec-Gpc": ["1"]}}, "duration": 0.004311695, "status": 500, "err_id": "ecnhjqf3b", "err_trace": "templates.(*Templates).executeTemplate (templates.go:315)"}

If I load 127.0.0.1/2020/05/10 (no trailing slash) I also get an error:

2020/11/01 14:55:04.661	ERROR	http.log.error	template: /templates/note.html:4:20: executing "/templates/note.html" at <include $markdownFilePath>: error calling include: open /Users/smerrill/skippy/public/2020/05/10.md: no such file or directory	{"request": {"remote_addr": "127.0.0.1:55270", "proto": "HTTP/1.1", "method": "GET", "host": "127.0.0.1", "uri": "/2020/05/10", "headers": {"Upgrade-Insecure-Requests": ["1"], "Dnt": ["1"], "Sec-Gpc": ["1"], "Connection": ["keep-alive"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:81.0) Gecko/20100101 Firefox/81.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"], "Accept-Language": ["en-US,en;q=0.5"], "Accept-Encoding": ["gzip, deflate"]}}, "duration": 0.001529981, "status": 500, "err_id": "h5qpkb66t", "err_trace": "templates.(*Templates).executeTemplate (templates.go:315)"}

I had expected try_files to rewrite the URI into /2020/05/10/index.md, which would then be served by the template.

Well I think the key is that you’re using .OriginalReq.URL.Path in your template. Rewrites will never affect this value. It’ll always be exactly what the browser sent. So instead, I think you should trigger redirects (with redir) to tell the browser to make a new request with the path you want.

You can use the long-form of try_files to do this, i.e. use the file matcher paired with a redirect instead of a rewrite.

     @directories {
       file {
         try_files {path}/
       }
     }
     redir @directories {http.matchers.file.relative}

Redirects 127.0.0.1/2020/05/10 to 127.0.0.1/2020/05/10/, but then the browser complains:

The page isn’t redirecting properly

Firefox has detected that the server is redirecting the request for this address in a way that will never complete.

I must have introduced a loop somehow. I guess that the try_files {path}/ is redirecting to /2020/05/10// which in turn redirects to /2020/05/10/// and so on?

I tried the above alongside

     @notes path_regexp \/\d{4}\/?.*$
     handle @notes {
         try_files {path} {path}.md {path}/index.md
         rewrite * /templates/note.html
     }

Both with and without the try_files line inside the @notes handler. Both produced the redirect problem.

Removing the directory redir bits, I replaced my note.html template with this:

{{ .OriginalReq }}
<br />
{{ .Req }}

And got this output:

{GET /2020/05/10 ...
{GET /templates/note.html HTTP/1.1 ...

This does not serve the index.md as I had desired.

I’d really like to keep the conditional processing in my templates to as little as possible. I don’t want to have templates that need to figure out “Is this a request for a directory or for a specific piece of content?” as that’s what index.md should be for. But I’m still unable to connect the pieces of the Caddyfile in the right way…

It’s entirely possible that I’m abusing the templating feature, and should go back to using Hugo to generate a completely static site. But if I can wrangle Caddy to work as expected, I’d really like to do so.

Thanks for your pointers so far. I appreciate it immensely. I’ll keep poking at this!

This topic was automatically closed after 30 days. New replies are no longer allowed.