Gzip, zstd, br priority order

[Forza]

1. The problem I’m having:

I wonder something. I was testing with Edge browser which sends Accept-Encoding: gzip, deflate, br, and I have encode zstd br gzip in my Caddyfile, why does Edge still receive gzip encoded content instead of brotli? Is it because Edge has br last in the list? I’d like Caddy to choose the encoding based on the order in Caddyfile.

2. Error messages and/or full log output:

3. Caddy version:

v2.7.6 => /usr/src/caddy/git/caddy@(devel)

xcaddy build --with github.com/caddyserver/caddy/v2=/usr/src/caddy/git/caddy  --with github.com/ueffel/caddy-brotli --with github.com/caddyserver/transform-encoder --with github.com/caddyserver/cache-handler --with github.com/kirsch33/realip --with github.com/git001/caddyv2-upload

d. My complete Caddy config:

## Main section
{
        # debug
        auto_https off
        log {
                output file /var/log/caddy/caddy_main.log {
                        roll_disabled
                }
                format json
        }
}
## Snippets
(main) {
        tls /etc/letsencrypt/live/{args[0]}/fullchain.pem /etc/letsencrypt/live/{args[0]}/privkey.pem {
                curves x25519 secp521r1 secp384r1 secp256r1
        }
        log {
                output file /var/log/caddy/{args[0]}_443.log {
                        roll_disabled
                }
                format json
        }
        encode zstd br gzip
}
(main80) {
        log {
                output file /var/log/caddy/{args[0]}_80.log {
                        roll_disabled
                }
                format json
        }
        encode zstd br gzip
}
## Hosts section
import vhosts/*.caddy

[francislavoie]

@Forza

github.com/ueffel/caddy-brotli

FYI this implementation is very slow.

If you must use Brotli, you should probably use GitHub - dunglas/caddy-cbrotli: A Caddy module providing support for the Brotli compression format. instead, which uses the C implementation of Brotli. It does mean you need to build Caddy with CGO_ENABLED=1 though, see the README.

@Forza

github.com/kirsch33/realip

You shouldn’t use this anymore. It’s replaced by trusted_proxies: Global options (Caddyfile) — Caddy Documentation.

@Forza

I wonder something. I was testing with Edge browser which sends Accept-Encoding: gzip, deflate, br, and I have encode zstd br gzip in my Caddyfile, why does Edge still receive gzip encoded content instead of brotli? Is it because Edge has br last in the list? I’d like Caddy to choose the encoding based on the order in Caddyfile.

I don’t have any brotli plugin installed. Can you run caddy adapt -p and show the encode handler config? It should have a prefer field that gets filled.

Basically, it should be respecting the order of encoders you list in your Caddyfile. The code is doing sorting according to “q-factor” (which your client is not setting, so that’s not relevant) then by prefer order.

caddyserver/caddy/blob/2c48dda109d213be0012743d8afb2d1f017ff878/modules/caddyhttp/encode/encode.go#L410

[Forza]

Thanks! I’m aware that the brotli encoding isn’t as fast, and CPU has been less of a bottle neck than bandwidth usage for me. For example file listings compress to half the size with br compared to gzip. But this was a while ago, so I’m grateful you pointed to the C brotli version. I’ll try it!

This is the json output for the encoding part.

"handle": [{
	"encodings": {
		"br": {
			"level": 1
		},
		"gzip": {
			"level": 1
		},
		"zstd": {}
	},
	"handler": "encode",
	"prefer": [
		"zstd",
		"br",
		"gzip"
	]
}]

[francislavoie]

That looks fine.

What if you put br first, does it change the result?

[Forza]

I have changed to c-brotli and also changed the order to br, zstd, gzip. Unfortunately it still chooses gzip in both Edge and Chrome.

Looking at the logs I see that brotli is stripped when accessing the site through my works computer (even though the dev-tools in the browser says brotli). This explains why the result is gzipped. When accessing the page via my phone I am correctly getting brotli compressed content.

Thanks for the heads up on the c-brotli and the trusted_proxies!

[francislavoie]

@Forza

Looking at the logs I see that brotli is stripped when accessing the site through my works computer (even though the dev-tools in the browser says brotli).

Huh? What’s doing that?

[Forza]

Corporate MITM firewall/proxy solution i guess.

[francislavoie]

I don’t see how, if you have a TLS connection to Caddy. The request is encrypted. If something tampered with it, it would break the connection and you would see a TLS error.

[Forza]

You are correct. But our company installs a certificate and so can alter/decrypt all traffic. Computers without this certificate cannot use internet at all.

[francislavoie]

Wow that’s awful. I’m sorry to hear that.

Yeah if they’re filtering traffic then it makes sense that they don’t support brotli, it’s not commonly supported yet.