We are using Caddy as a load balancer (three nodes), but one of the features of the product we are using is “document attachment” (basically, you pick a file and the site uploads it). The problem is that when we go to pick the file, it posts a 404 not found, and when we attempt to attach it, it tries three times and posts 400 errors each time. The 400 errors all say something about invalid ETags. We contacted support for the product and they said that our load balancer (Caddy) is changing the case of the ETag and because they do strict validation on the headers, it’s throwing an error. We have tried disabling HTTP2 and header upstream ETag (pass through), but so far nothing has worked. Any ideas would be welcome. Thanks in advance.
Is Caddy serving the files directly off disk, like a CDN, or just proxying to your vendor’s product?
When you say Caddy is changing the case of the ETag, do you mean the actual letter case, like between upper case and lower case?
Caddy is just proxying to the three application nodes (load balancing). Yes, the vendor’s development team (relayed to us through tech support) said that our load balancer was changing the case (i.e. text–UPPER and lower) of the ETag. Apparently, they haven’t bothered to follow the HTTP/2 spec and do their validations case-insensitively, so they threw the issue back in our lap.
Caddy doesn’t seem to do any modification of text case. Did they qualify their statement with any evidence of this happening?
For a quick test:
whitestrake at apollo in ~/Projects/test
❯ caddy -version
Caddy 0.11.0 (non-commercial use only)
whitestrake at apollo in ~/Projects/test
❯ cat Caddyfile
:2015 {
status 200 /
header / Etag "lowerUPPER"
}
:2016 {
proxy / :2015
}
whitestrake at apollo in ~
❯ curl -i localhost:2015
HTTP/1.1 200 OK
Etag: lowerUPPER
Server: Caddy
Date: Wed, 31 Oct 2018 23:32:53 GMT
Content-Length: 0
whitestrake at apollo in ~
❯ curl -i localhost:2016
HTTP/1.1 200 OK
Content-Length: 0
Date: Wed, 31 Oct 2018 23:32:54 GMT
Etag: lowerUPPER
Server: Caddy
Server: Caddy
For reference, the header manipulation undertaken by a Caddy proxy is here in the proxy code:
The actual function doing the manipulation of headers is here - as can be seen, it only performs any operations on headers which have rules configured via the Caddyfile, i.e. header_upstream and header_downstream.
The only other thing I can find is that while sending the request to the backend, Caddy strips out the Connection and hop-by-hop headers.
What OS are you using for all load balancers?
What product that isn’t working are you using?
Which version of Caddy are you using?
Could you please share your full caddyfile?
Are you using gzip?
gzip seems to write weak etags. I would try setting it off if you’re using it:
https://github.com/mholt/caddy/blob/master/caddyhttp/gzip/gzip.go#L124
2 Likes
You got it! I turned off gzip and it started working. Huge thanks (you just saved me a lot of time)! And thanks to everyone else who replied as well 
1 Like
@matt should a note about it be added on one of these pages?
https://caddyserver.com/docs/proxy
https://caddyserver.com/docs/gzip
1 Like
Yes, good idea – I will add it to the gzip page. Thanks for investigating!
1 Like
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.