Regarding on-the-fly compression and precompressed files

Help
1

1. The problem I’m having:

tl;dr
Since the encode directive runs after try_files but before file_server, will adding precompressed gzip zstd whatever directive to file_server create double duty for Caddy, where it encodes files found by try_files on-the-fly but then file_server finds a sidecar anyway and serves it instead?

long version:
I have a WordPress site block in Caddyfile with directives to serve cached HTML pages (named https-index.html) directly. The cache files have sidecars (https-index.html.gz and https-index.html.zst), and I want to know if the site block in my Caddyfile is the best way to serve precompressed files by bypassing on-the-fly compression.

The three directives relevant to this topic are try_files, file_server, and encode. As per Caddy’s documentation, the order of execution of these directives are:

… → try_files → … → encode → … → file_server → …

By this logic, any file found by try_files will be encoded before file_server sends it to the client. If I define precompressed gzip whatever in file_server, it will look for a sidecar and serve it if found.

I have noticed that precompressed under file_server takes precedence over encode, so I wonder if Caddy automatically skips encoding files that have a precompressed sidecar. Also, what’s the overhead of looking for precompressed files vs encoding everything on the fly (which one would be faster under very heavy load)?

2. Error messages and/or full log output:

n/a

3. Caddy version:

2.8.4

4. How I installed and ran Caddy:

Caddy’s official Debian repository

a. System environment:

Debian 12.6 x64

b. Command:

n/a

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

example.com {
	root * /path/to/site/files
	
	encode zstd gzip
	
	file_server {
		precompressed zstd gzip
	}
	
	@staticFiles {
		path_regexp .*\.(jpg|jpeg|png|gif|ico|css|js|woff|woff2|webp|jxl|avif|mp4|webm)$
	}
	header @staticFiles {
		Cache-Control "public, max-age=31536000, stale-while-revalidate=600, stale-if-error=120"
	}
	
	header {
		-Server

		Strict-Transport-Security "max-age=31536000; includeSubDomains; preload"
		X-Content-Type-Options "nosniff"
		# X-Frame-Options "SAMEORIGIN"
		Referrer-Policy "strict-origin-when-cross-origin"
		Permissions-Policy "geolocation=(), microphone=(), camera=(), payment=(), usb=(), fullscreen=(self)"
		X-XSS-Protection "1; mode=block"
		Content-Security-Policy "default-src 'self'; script-src 'self' 'unsafe-inline' 'unsafe-eval' blob:; style-src 'self' 'unsafe-inline'; img-src 'self' data:; font-src 'self' data:; object-src 'none'; frame-ancestors 'self'; upgrade-insecure-requests;"
	}
	
	log {
		output file /path/to/access.log {
		        roll_size 10MiB
		        roll_keep 10
		        roll_keep_for 1440h
		}
	}
	
	@disallowed {
		path /xmlrpc.php /wp-config.php /.* *.sql
	}
	respond @disallowed "403 access denied" 403

	@cache {
		not header_regexp Cookie "comment_author|wordpress_[a-f0-9]+|wp-postpass|wordpress_logged_in|woocommerce_items_in_cart|wp_woocommerce_session"
		not path_regexp "(/wp-admin/|/xmlrpc.php|/wp-(app|cron|login|register|mail).php|wp-.*.php|/feed/|index.php|wp-comments-popup.php|wp-links-opml.php|wp-locations.php|sitemap(index)?.xml|[a-z0-9-]+-sitemap([0-9]+)?.xml)"
		not method POST
		not expression {query} != ''
	}

	route @cache {
		try_files /wp-content/cache/cache-enabler/{host}{uri}/https-index.html {path} {path}/index.php?{query}
	}
	
	php_fastcgi unix//run/php/php8.1-fpm.sock
}
2

No. The encode directive is ordered the way it is to prepare a response writer wrapper before something after it tries to write the response. The HTTP middleware chain is followed down, then back up. The repsonse is encoded (as a stream) on the way back up the middleware chain, after file_server has selected a file and is writing the response. If file_server servers a pre-compressed file, then it writes an HTTP header Content-Encoding which the encode handler will see and recognize that as a signal to skip doing its own encoding.

So in practice, this is what happens (assuming only those three handlers run, but obviously there is probably more of them at play):

  • try_files rewrites the URL, passes to the next handler
  • encode sets up a response writer wrapper, passes
  • file_server selects a file, reads it from disk, starts writing it
  • file_server returns, meaning it doesn’t pass onto the next handler, it hands control back to the previous handler
  • encode now processes the response, considering HTTP response headers already written, deciding to encode or not, if it encodes it will also add its own response headers as well
  • try_files technically gets control, but does nothing because it’s already done its job
  • now the chain is done and the response is written out

It’s clearer if you see the code. All HTTP handlers in Caddy look like this:

func (m Middleware) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
	// run some code on the way DOWN the middleware chain
	
	err := next.ServeHTTP(w, r)
	if err != nil {
		return err
	}

	// run some code on the way back UP the middleware chain

	return nil
}

Some (most) may just look like this if they only need to do stuff on the way down but not up:

func (m Middleware) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
	// run some code on the way DOWN the middleware chain	
	return next.ServeHTTP(w, r)
}

If a handler just doesn’t call next (and just does return nil when there’s no error) then it’s considered “terminal”, i.e. it terminates the chain, will prevent any handlers configured (ordered) after it from running. That’s the case for file_server by default since writing a response means nothing else can also write a response (but something could filter/process the response, like encode on the back way up).

Precompressed will be much faster because it doesn’t need to spend any CPU resources on compression.

3 Likes
3

Thank you tons for the detailed explanation!

1 Like
4

Great answer. :clap:

2 Likes
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.