Caddy 0.10.9 + Varnish Cache 5.2 HTTP/2 thread starvation bug

@mwpastore posted his Caddy + Varnish Cache configuration and results here which looked good for non-HTTP/2 based HTTPS load tests with wrk load tester.

So I thought I’d check them out for HTTP/2 based HTTPS loads via h2load HTTP/2 HTTPS load testing tool. However, the results aren’t that great probably due to some bugs with Varnish Cache 5.2 and HTTP/2 issues bugs. My full results are at https://community.centminmod.com/threads/caddy-http-2-server-benchmarks-part-2.12873/#post-54821.

Varnish 5.2 Bugs

Tests

Testing Caddy 0.10.9 GCC 7.1.1 build binary with Varnish Cache 5.2 where Caddy HTTPS frontend acts as a SSL proxy for SSL termination (port 443) to Varnish Cache backend (port 6081) and then Varnish Cache talks to Caddy HTTP (port 80) as backend.

curl -I https://caddybench.domain.com
HTTP/2 200
accept-ranges: bytes
age: 6
content-type: text/html; charset=utf-8
date: Thu, 21 Sep 2017 01:09:21 GMT
etag: W/"owd8wg4pb"
last-modified: Sat, 16 Sep 2017 09:26:40 GMT
server: Caddy
server: Caddy
vary: Accept-Encoding
via: 1.1 varnish (Varnish/5.2)
x-cache: HIT
x-cache-hits: 1
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-powered-by: Caddy via CentminMod
x-varnish: 32770 3
x-xss-protection: 1; mode=block

Results

7x runs of h2load test with parameters:

h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443

Avg: 145.57 req/s with only 31.99% completion of requests on average

echo "Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test + Varnish 5.2"; sleep 30; for i in {1..7}; do /usr/local/bin/h2load -t2 -c200 -m100 -n10000 -H 'Accept-Encoding: gzip' https://caddybench.domain.com:443 > h2load.$i.caddy.log; cat h2load.$i.caddy.log | awk '/finished in/ {print $4 " req/s "} /requests: / {print ($8/$2*100)"% completed"}' | tr -d '\n'; echo; sleep 30; done;
Caddy 0.10.9 Source GCC 7.1.1 Build HTTP/2 h2load stress test + Varnish 5.2
699.98 req/s 94.49% completed
166.57 req/s 46.03% completed
146.46 req/s 59.46% completed
56.99 req/s 23.93% completed
0.00 req/s 0% completed
0.00 req/s 0% completed
0.00 req/s 0% completed

This is compared to other tests I have run here.

Comparison table

caddy-vs-nginx-h2load-http2-loadtests-varnish-210917-01

Summary

So for folks looking at using Varnish Cache with Caddy, make sure to test both HTTP/1.1 and HTTP/2 based HTTPS load tests using a proper load testing tool like h2load and not just test HTTP/1.1 HTTPS loads with the usual wrk, siege, apachebench tools etc as well as keep an eye on Varnish Cache HTTP/2 related bugs.

3 Likes

Any better solution? How is built in cache proxy for wordpress or ghost? How about HAProxy?

You mean for Caddy http.cache proxy caching ? @Whitestrake advises against it Htttp.cache plugin usage - #2 by Whitestrake but probably because the http.cache plugins needs a few more features like cache or exclude from cache based on key or cookies so you can further fine tune what to cache or not within wordpress.

Haproxy is load balancer not really a HTTP/HTTPS web accelerator like Varnish Cache or Caddy’s http.cache plugin.

Varnish Cache requires a ssl terminator in front of it, this can be handled with haproxy, nginx or caddy.

Have you take a look into http://trafficserver.apache.org/ ?

why would you want to use that in what capacity with caddy pairing ?