[SOLVED] How to use file_server precompressed?

Hi I am looking at this documentation in file_server:

  • precompressed is the list of encoding formats to search for precompressed sidecar files. Arguments are an ordered list of encoding formats to search for precompressed sidecar files. Supported formats are gzip, zstd and br.

I do not know what a sidecar file is, I am assuming it is a static file?

I am trying to figure out how to make this work:

Caddyfile:

:1111 {
  root * /home/brucejo/temp/caddy/precompressed
  file_server {
    precompressed gzip
  }
}
~/temp/caddy/precompressed$ echo "Hello World" > index.html
~/temp/caddy/precompressed$ gzip -c > index.html.gz
~/temp/caddy/precompressed$ caddy reload
~/temp/caddy/precompressed$ curl localhost:1111
Hello World
~/temp/caddy/precompressed$ mv index.html index.html.FOO
~/temp/caddy/precompressed$ curl localhost:1111
~/temp/caddy/precompressed$ curl localhost:1111/index.html.gz
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

I tried using rewrite {path} {path}.gz but it apparently does not take placeholders? Error output: reload: adapting config using caddyfile: parsing caddyfile tokens for 'rewrite': Caddyfile:3 - Error during parsing: Wrong argument count or unexpected line ending after '{http.request.uri.path}.gz'

So then I did a uri replace {path} {path}.gz.

~/temp/caddy/precompressed$ curl localhost:1111/index.html
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.

It rewrites, but it does not know the content is precompressed.

I assume I am missing a key ingredient to be able to use precompressed, help please?

Thanks!

Does index.html need to be there so that Caddy can know to pick up caddy.html.gz? Does sidecar file mean they both need to be there?

thanks.

The idea is file_server itself will look for a file matching the one requested with any of the given compression algorithms you list as supported as the suffix, if the client signaled that it supports that compression (with the Accept-Encoding header). So with curl, you’ll need to make sure to use the --compressed option to make it make a request with Accept-Encoding with the ones curl supports.

So, for a request like /index.html, with precompressed gzip turned on, Caddy will look at the headers, if it sees gzip as an accepted encoding, then it will look for /index.html.gz on disk, and if it finds that it’ll serve that file as-is, but also with the Content-Encoding header set, telling the client that “yeah, I am sending compressed content”, so the client can then decompress it and show you.

So try running curl -v --compressed localhost:1111/index.html and see what happens.

A “sidecar file” is just a file beside another in this case, like a file that’s paired with another. So index.html.gz is a sidecar to index.html

Btw, Caddy has the encode gzip directive, so you could just let Caddy compress things on-the-fly. It’s usually good enough, gzip compression is very fast and has very little overhead. If you want maximum compression (smallest files) then you can precompress with brotli (assuming the client supports brotli decompression).

1 Like

Thanks for the response @francislavoie,

To be clear, the index.html file does not need to be there?

I have these files in my root:

Caddyfile  index.html.FOO  index.html.gz 

It still does not seem to work (if I do not need both index.html & index.html.gz in the root).

What I get using curl
~/temp/caddy/precompressed$ curl -v --compressed localhost:1111/index.html
*   Trying 127.0.0.1:1111...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 1111 (#0)
> GET /index.html HTTP/1.1
> Host: localhost:1111
> User-Agent: curl/7.68.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: Caddy
< Date: Thu, 31 Mar 2022 15:45:53 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact

Requesting index.html.gz works…

~/temp/caddy/precompressed$ curl -v --compressed localhost:1111/index.html.gz
*   Trying 127.0.0.1:1111...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 1111 (#0)
> GET /index.html.gz HTTP/1.1
> Host: localhost:1111
> User-Agent: curl/7.68.0
> Accept: */*
> Accept-Encoding: deflate, gzip, br
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Content-Length: 43
< Content-Type: application/gzip
< Etag: "r9l7gb17"
< Last-Modified: Thu, 31 Mar 2022 02:12:11 GMT
< Server: Caddy
< Date: Thu, 31 Mar 2022 15:49:33 GMT
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 43)
* Closing connection 0

Also, when I enter localhost:1111/index.html in the browser I get page can’t be found, even though Accept-Encoding includes gzip

Request Header
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Host: localhost:1111
Pragma: no-cache
sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="99", "Google Chrome";v="99"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36

When I enter localhost:1111/index.html.gz in the browser the file is downloaded, which I think is reasonable.

Yeah, the original file still needs to exist – Caddy with look there first, and only look for a sidecar if compression is on.

The idea is that otherwise, clients which don’t have compression support wouldn’t have a way to get the data at all.

Thanks @francislavoie,

Is there any way I can verify that Caddy is picking up the .gz file? The logs really do not say…

2022/03/31 16:37:19.351 info    http.log.access.log0    handled request {"request": {"remote_addr": "127.0.0.1:40450", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:1111", "uri": "/index.html", "headers": {"User-Agent": ["curl/7.68.0"], "Accept": ["*/*"], "Accept-Encoding": ["deflate, gzip, br"]}}, "common_log": "127.0.0.1 - - [31/Mar/2022:09:37:19 -0700] \"GET /index.html HTTP/1.1\" 200 43", "user_id": "", "duration": 0.0072198, "size": 43, "status": 200, "resp_headers": {"Server": ["Caddy"], "Content-Encoding": ["gzip"], "Vary": ["Accept-Encoding"], "Etag": ["\"r9l786c\""], "Content-Type": ["text/html; charset=utf-8"], "Last-Modified": ["Thu, 31 Mar 2022 02:07:18 GMT"], "Accept-Ranges": ["bytes"]}}
2022/03/31 16:39:29.692 info    http.log.access.log0    handled request {"request": {"remote_addr": "[::1]:41388", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:1111", "uri": "/index.html", "headers": {"Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "If-Modified-Since": ["Thu, 31 Mar 2022 02:07:18 GMT"], "Sec-Fetch-Mode": ["navigate"], "Sec-Fetch-User": ["?1"], "Sec-Fetch-Dest": ["document"], "If-None-Match": ["\"r9l786c\""], "Connection": ["keep-alive"], "Sec-Ch-Ua": ["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"99\", \"Google Chrome\";v=\"99\""], "Sec-Fetch-Site": ["none"], "Accept-Language": ["en-US,en;q=0.9"], "Cache-Control": ["max-age=0"], "Sec-Ch-Ua-Mobile": ["?0"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"], "Accept-Encoding": ["gzip, deflate, br"]}}, "common_log": "::1 - - [31/Mar/2022:09:39:29 -0700] \"GET /index.html HTTP/1.1\" 304 0", "user_id": "", "duration": 0.0002778, "size": 0, "status": 304, "resp_headers": {"Server": ["Caddy"], "Content-Encoding": ["gzip"], "Vary": ["Accept-Encoding"], "Etag": ["\"r9l786c\""]}}

I think I answered my own question.

In the browser the response header has Content-encoding: gzip even though I have no encode directive in the Caddyfile.

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.