I can confirm that this behaviour also happens to me with 2.4.0 and default config. Running on Linux Ubuntu 20.04
docker run -d -p 80:80 caddy:2.4.0
curl -I localhost:80/index.html
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 12226
Content-Type: text/html; charset=utf-8
Last-Modified: Tue, 11 May 2021 01:04:23 GMT
Server: Caddy
Date: Fri, 14 May 2021 13:19:20 GMT
curl -I localhost:80/INDEX.HTML
HTTP/1.1 404 Not Found
Server: Caddy
Date: Fri, 14 May 2021 13:19:27 GMT
It looks like Caddy is case sensitive. And this info: Paths with uppercase letters aren't working - #3 by matt does not apply here. @matt Could you confirm this?