I am trying to troubleshoot a problem with caddy that I am encountering. On why certain clients return “403 Forbidden” and not others and how to troubleshoot these problems as default log files are not sufficient. When I use a very similar setup with apache I do not have issues using same server certificates.
Configuration is using recent (August) mostly stock Caddy with http.proxy running under Windows 7. I control the complete server and client application sides of the connection. I don’t use transparent on the proxy as it messes up forwarding on localhost to apps by replacing ip address with hostname in the request. Anyway that is probably not relevant since it doesn’t seem to be used.
The server application on other end of proxy doesn’t matter as I can verify that it is not using the internal application port (using rawcap, netstat, tcpview) so the 403 is not from the application. I’ve tried to use wireshark but it uses elliptic curve encryption which defeats that so then tried fiddler but there is nothing obvious and they just show the failure response. Wireshark on the server side doesn’t show much but traffic from working and failed clients seem mostly the same so doesn’t look like different certificates are being used and both are selecting same EC encryption method.
The CA is internal self-signed and clients have client key from same ca (though no way to force caddy to require that yet). When I use curl and openssl I can try to use those but doesn’t matter. Also problem occurs the same on all browsers, postman, curl, openssl s_client so I tend to believe this is caddy or go serving up the error but I cannot tell why.
Most machines that I test with can call this endpoint and get the simple response without issue as expected. But I have one machine that cannot complete the call. I don’t have control over the machine and I suspect there may be ssl mitm hardware or software around the machine’s network but hard to say as IT will not respond to inquiries. Not that it matters as I would still like to understand how to troubleshoot this kind of issue.
Access log shows something like the following where the first statement is the failure from bad client and second is from my dev machine. Error log shows nothing.
xxx.xxx.xxx.xxx - - [25/Sep/2017:15:24:18 +0000] "GET /path/test HTTP/1.1" 403 38
xxx.xxx.xxx.xxx - - [25/Sep/2017:15:24:21 +0000] "GET /path/test HTTP/1.1" 200 34
I tried the errors visible setting but doesn’t do anything from what I can see.
So is there any better way to troubleshoot this kind of problem?
The config file looks approximately like the following with path and ports changed:
:443
gzip
log logs/access.log {
rotate_size 10 # Rotate after 10 MB
rotate_age 90 # Keep rotated files for 90 days
rotate_keep 20 # Keep at most 20 log files
rotate_compress # Compress rotated log files in gzip format
}
errors logs/error.log {
rotate_size 10 # Rotate after 10 MB
rotate_age 90 # Keep rotated files for 90 days
rotate_keep 20 # Keep at most 20 log files
rotate_compress # Compress rotated log files in gzip format
}
tls conf/ssl/server.crt conf/ssl/server.key {
ca conf/ssl/caroot.pem
}
proxy /path 127.0.0.1:12345 {
header_upstream X-Real-IP {remote}
header_upstream X-Forwarded-Host {host}
header_upstream X-Forwarded-For {remote}
header_upstream X-Forwarded-Proto {scheme}
}