When does Caddy auto add Vary Accept-Encoding header?

1. Caddy version (caddy version):

v2.0.0 h1:pQSaIJGFluFvu8KDGDODV8u4/QRED/OPyIR+MWYYse8=

2. How I run Caddy:

Installed Caddy via copr YUM repo

yum-config-manager --add-repo https://copr.fedorainfracloud.org/coprs/g/caddy/caddy/repo/epel-7/group_caddy-caddy-epel-7.repo
yum -y install caddy --disablerepo=epel
sed -i 's|:80|http://caddy.domain.com:81\n\nheader x-powered-by "caddy centminmod"\nheader vary Accept-Encoding\nencode gzip\n|' /etc/caddy/Caddyfile
caddy trust
cp -a /usr/share/caddy/index.html /usr/share/caddy/caddy-index.html
service caddy start
service caddy status
chkconfig caddy on

a. System environment:

  • CentOS 7.8
  • Nginx running on port 80 and 443
  • VirtualBox on Windows 10 Pro

b. Command:

service caddy start

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
    http_port   81
    https_port  4444
}

http://caddy.domain.com:81 {

    header x-powered-by "caddy centminmod"
    header vary Accept-Encoding
    header X-Content-Type-Options nosniff
    header X-XSS-Protection "1; mode=block"
    encode gzip
    
    # Set this path to your site's directory.
    root * /usr/share/caddy
    
    # Enable the static file server.
    file_server
    
    # Or serve a PHP site through php-fpm:
    # php_fastcgi localhost:9000
    log {
        output file /var/log/caddy/access_http.log {
            roll_size 100mb
            roll_keep 10
            roll_keep_for 720h
        }
        format single_field common_log
        level INFO
    }
}

https://caddy.domain.com:4444 {

    tls internal
    header x-powered-by "caddy centminmod"
    header vary Accept-Encoding
    header X-Content-Type-Options nosniff
    header X-XSS-Protection "1; mode=block"
    encode gzip
    
    # Set this path to your site's directory.
    root * /usr/share/caddy
    
    # Enable the static file server.
    file_server
    
    # Or serve a PHP site through php-fpm:
    # php_fastcgi localhost:9000
    log {
        output file /var/log/caddy/access_https.log {
            roll_size 100mb
            roll_keep 10
            roll_keep_for 720h
        }
        format single_field common_log
        level INFO
    }
}

3. The problem I’m having:

In my web browser I am seeing duplicate Vary Accept-Encoding headers. But not seeing such duplicate headers on curl command line ?

On curl command line I see

curl -Ik https://caddy.domain.com:4444/caddy-index.html
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 17:38:15 GMT
curl -Ik -H "Accept-Encoding:gzip" https://caddy.domain.com:4444/caddy-index.html
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 17:38:28 GMT

But within web browser for same requested file I see duplicate headers

If I remove specific header for Vary in Caddyfile, the curl headers have Vary missing while browser adds Vary header

curl -Ik https://caddy.domain.com:4444/caddy-index.html                          
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 17:42:19 GMT
curl -Ik -H "Accept-Encoding:gzip" https://caddy.domain.com:4444/caddy-index.html
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 17:42:26 GMT

image

Vary: Accept-Encoding is added if you use the encode directive.

Only for web browsers though and not curl header checks ?

When you used curl, it didn’t actually gzip the response. I think you have something missing in your command for it to actually gzip it.

Maybe you need the space after the colon? -H "Accept-Encoding: gzip"

ah you’re right in that Caddy isn’t gzip compressing the response, with Vary header removed I see

curl -Ik -H "Accept-Encoding: gzip" https://caddy.domain.com:4444/caddy-index.html 
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 18:01:38 GMT

it’s working on Nginx via curl

curl -Ik -H "Accept-Encoding: gzip" https://ngx.domain.com/caddy-index.html
HTTP/1.1 200 OK
Date: Fri, 08 May 2020 18:02:58 GMT
Content-Type: text/html; charset=utf-8
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: W/"5eb30579-2fc2"
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Encoding: gzip

so my Caddyfile above with encode gzip isn’t working ?

Well it’s clearly working in your browser, there must be some piece missing when trying via curl. :thinking:

What are the full request headers in your browser?

Yeah it seems specific to Caddy and curl. I tested with h2load and my forked wrk tools and Accept-Encoding gzip return gzip responses if you look at data transfer size and size per request

caddy

h2load 51.76KB /10 requests = 5.176KB per request

h2load -t1 -c1 -n10 -m1 -H "Accept-Encoding: gzip" https://caddy.domain.com:4444/caddy-index.html
starting benchmark...
spawning thread #0: 1 total client(s). 10 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES256-GCM-SHA384
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 35.84ms, 279.04 req/s, 1.41MB/s
requests: 10 total, 10 started, 10 done, 10 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 51.76KB (53002) total, 277B (277) headers (space savings 89.82%), 51.17KB (52400) data
                     min         max         mean         sd        +/- sd
time for request:     1.07ms      7.95ms      3.09ms      2.47ms    80.00%
time for connect:     3.30ms      3.30ms      3.30ms         0us   100.00%
time to 1st byte:     5.17ms      5.17ms      5.17ms         0us   100.00%
req/s           :     285.15      285.15      285.15        0.00   100.00%

wrk 7411 requests in 6.01s, 39.69MB read = 5.48KB per request

wrk -t1 -c1 -d6s --breakout --latency -H "Accept-Encoding: gzip" https://caddy.domain.com:4444/caddy-index.html
Running 6s test @ https://caddy.domain.com:4444/caddy-index.html
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency     0.86ms  608.49us   9.98ms   95.56%
    Connect   813.00us    1.15ms   2.44ms    0.00%
    TTFB      788.86us  581.25us   9.80ms   96.11%
    TTLB       72.67us  150.94us   5.38ms   98.29%
    Req/Sec     1.24k   192.50     1.76k    73.33%
  Latency Distribution
     50%  742.00us
     75%    0.87ms
     90%    1.18ms
     99%    4.03ms
  7411 requests in 6.01s, 39.69MB read
Requests/sec:   1233.90
Transfer/sec:      6.61MB

nginx

h2load 52.09KB /10 requests = 5.209KB per request

h2load -t1 -c1 -n10 -m1 -H "Accept-Encoding: gzip" https://ngx.domain.com/caddy-index.html
starting benchmark...
spawning thread #0: 1 total client(s). 10 total requests
TLS Protocol: TLSv1.2
Cipher: ECDHE-ECDSA-AES128-GCM-SHA256
Server Temp Key: ECDH P-256 256 bits
Application protocol: h2
progress: 10% done
progress: 20% done
progress: 30% done
progress: 40% done
progress: 50% done
progress: 60% done
progress: 70% done
progress: 80% done
progress: 90% done
progress: 100% done

finished in 8.26ms, 1211.39 req/s, 6.16MB/s
requests: 10 total, 10 started, 10 done, 10 succeeded, 0 failed, 0 errored, 0 timeout
status codes: 10 2xx, 0 3xx, 0 4xx, 0 5xx
traffic: 52.09KB (53339) total, 2.02KB (2070) headers (space savings 26.86%), 49.84KB (51040) data
                     min         max         mean         sd        +/- sd
time for request:      396us       672us       508us        94us    60.00%
time for connect:     2.48ms      2.48ms      2.48ms         0us   100.00%
time to 1st byte:     3.00ms      3.00ms      3.00ms         0us   100.00%
req/s           :    1243.22     1243.22     1243.22        0.00   100.00%

wrk 10579 requests in 6.00s, 55.50MB read = 5.37KB per request

wrk -t1 -c1 -d6s --breakout --latency -H "Accept-Encoding: gzip" https://ngx.domain.com/caddy-index.html
Running 6s test @ https://ngx.domain.com/caddy-index.html
  1 threads and 1 connections
  Thread Stats   Avg      Stdev     Max   +/- Stdev
    Latency   625.91us  722.06us  15.61ms   98.34%
    Connect     1.12ms  516.88us   2.06ms   60.00%
    TTFB      535.41us  449.27us  11.48ms   98.65%
    TTLB       90.43us  574.72us  15.22ms   99.45%
    Req/Sec     1.77k   302.23     2.30k    65.00%
  Latency Distribution
     50%  552.00us
     75%  615.00us
     90%  729.00us
     99%    2.89ms
  10579 requests in 6.00s, 55.50MB read
Requests/sec:   1762.80
Transfer/sec:      9.25MB

So h2load and wrk tools with -H "Accept-Encoding: gzip" are returning gzip compressed responses from nginx and caddy. But caddy isn’t returning gzip compressed response when -H "Accept-Encoding: gzip" is set for curl requests ?

edit: i updated 1st post with caddy version/environment info too to make it easier.

Ah right - curl -I is the HEAD method. There’s nothing to gzip on HEAD. If you make a GET request it’ll be gzipped.

For next time, best to use curl -v, we get more info that way (request headers are printed)

1 Like

Ah interesting though HEAD request worked for Nginx and not Caddy. But get request does work for Caddy

Caddy GET request

curl -skD - -H "Accept-Encoding: gzip" https://caddy.domain.com:4444/caddy-index.html -o /dev/null
HTTP/1.1 200 OK
Content-Encoding: gzip
Content-Type: text/html; charset=utf-8
Etag: "q9xapl9fm"
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Server: Caddy
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Powered-By: caddy centminmod
X-Xss-Protection: 1; mode=block
Date: Fri, 08 May 2020 19:09:02 GMT
Transfer-Encoding: chunked

Nginx GET request

curl -skD - -H "Accept-Encoding: gzip" https://ngx.domain.com/caddy-index.html -o /dev/null                                              
HTTP/1.1 200 OK
Date: Fri, 08 May 2020 19:10:16 GMT
Content-Type: text/html; charset=utf-8
Last-Modified: Wed, 06 May 2020 18:44:09 GMT
Transfer-Encoding: chunked
Connection: keep-alive
Vary: Accept-Encoding
ETag: W/"5eb30579-2fc2"
Server: nginx centminmod
X-Powered-By: centminmod
X-Xss-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Content-Encoding: gzip

Not sure if it’s something to fix in Caddy for other folks who might test with a curl HEAD request and think Caddy is broken compared to other web servers ?

I mean, HEAD has no response body so it doesn’t make sense to gzip.

1 Like

Ah indeed gzip encoding doesn't seem to work · Issue #3333 · caddyserver/caddy · GitHub though Caddy should document this somewhere as nginx, apache, h2o and litespeed servers all do show gzip responses for HEAD for diagnostic purposes.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.