Matching /admin with /filesystem/admin/index.<md4>.html

1. Output of caddy version:

v2.5.2 h1:eCJdLyEyAGzuQTa5Mh3gETnYWDClo1LjtQm2q9RNZrs=

2. How I run Caddy:

docker run -d -p 80:80 -v caddy_data:/data caddy

a. System environment:

Latest docker.

b. Command:

n/a

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

:80 {
        root * /usr/share/caddy
        file_server
}

3. The problem I’m having:

We are producing static websites on a central CICD server and hosting each site on multiple CDN nodes.

The CDN is in-house: Each node is a Caddy server and this is a “push CDN” such that we cannot rely upon a typical origin pull or TTL cache invalidation strategy.

Our static site generator is Hugo and Hugo does not support partial builds (unless you enable persistent build caches which is impractical in most ephemeral, container-based, cloud CICD environments).

Builds/updates are high traffic and hundreds of builds are running at any one time.

As such, when we push the latest build from the CICD server out to all CDN nodes, if we wish to do this in a bandwidth-efficient fashion, we must select a file sync diffing strategy that simultaneously:

  1. Ignores file ‘created’ timestamps; and
  2. Can distinguish between changed and unchanged files of the same size and name.

We’re heading into checksum territory.

We are thus exploring various local/remote strategies that are not related to Caddy (and I won’t waste the forum’s time with those here) and one that might work with Caddy’s help.

So far I have not found a way to configure this strategy, it may not be possible, but I thought I’d try my luck here with the awesome Caddy community! Cannot tell you enough how much we love Caddy :slight_smile: !!!

The strategy proposed is:

  1. During CICD add a checksum of the file’s contents (md4, etc) to the filename of every file:
/index.<checksum>.html
/admin/index.<checksum>.html
/assets/app.<checksum>.js
/assets/brand.<checksum>.css
etc
  1. Push/sync these files to each CDN node using filename diffs.

  2. Configure Caddy to try_file for (eg.):

/index.<regex-or-wildcard>.html
....when the request is to....
/

/admin/index.<regex-or-wildcard>.html
....when the request is to....
/admin

…and so forth.

This would only be required for the html files. The /assets can all remain fingerprinted in the html that calls them.

4. Error messages and/or full log output:

n/a

5. What I already tried:

I have tried permutations of file_server, try_files, named matchers (esp. file), url, path_regexp and so on.

The common challenge is that all of these accommodate a regex/wildcard for requests/paths, but do not seem to accommodate regex/wildcards for file names on the caddy file system.

I got a little excited when I saw the configuration for…

‘try_policy’: ‘first_exist|smallest_size|largest_size|most_recent_modified’

…because that hints at rules to select from multiple matching files on the filesystem, but it doesn’t seem to be the filesystem wildcard feature that I’m looking for.

I’m completely expecting to be told that this cannot, or should not, be done.

These are not the droids you are looking for.

But…just maybe…

Any advice greatly appreciated!

6. Links to relevant resources:

File Matchers

Thx in advance for your advice Caddy community.

1 Like

I don’t think it’s possible right now, but I think it could be done.

I think it would essentially mean somekind of first_glob policy; would be the same as first_exist but using filepath.Glob() to match for files instead, returning the first result from that (if there’s multiple results then… result is kinda undefined).

That said, if you need it now, you’d probably be better off writing a Caddy plugin to do the rewrite you need, which should be quite easy given your requirements.

If you want to sponsor the development of this feature, please consider sponsoring Matt so he can prioritize it:

4 Likes

Thanks @francislavoie for the super fast and super helpful response, greatly and genuinely appreciated.

Yes I can see now how extending try_policy or a plugin could get us there. We’re assessing the non-caddy approaches in parallel and if those are all too hard then its the perfect excuse for us to write our first Caddy plugin and share it with the community.

We’d also loooove to sponsor @matt and have already discussed this internally. Despite having a lot of activity, we’re still an early stage startup and thus can’t sponsor anyone just yet BUT we have a clause in our Charter to sponsor several open source projects that we rely upon (including Caddy!) once we reach a revenue trigger level. So wish us luck!

2 Likes

Sounds great. Yeah, we could prioritize this for a sponsor; right now I’m trying to get 2.6 released. We’d love to have you onboard as a sponsor, so let me know what you need or what questions you have when you are ready!

1 Like

@jrock If you have a sec could you open an issue and link to this thread, so we/I don’t forget? Sign in to GitHub · GitHub

Thanks!

As always, very much looking forward to what goodness 2.6 brings to Caddy.

Yes will absolutely explore sponsorship with you as we approach target.

OK, I broke out the Star Wars quotes above already so…

Stay on target

2 Likes

I appreciate the Star Wars quote.

1 Like

Done!

Glob matcher for file names on the filesystem #4988

2 Likes

Thanks.

Implemented in:

This topic was automatically closed after 30 days. New replies are no longer allowed.