Troubleshoot 403 Forbidden with proxy


(Theo) #1

I am trying to troubleshoot a problem with caddy that I am encountering. On why certain clients return “403 Forbidden” and not others and how to troubleshoot these problems as default log files are not sufficient. When I use a very similar setup with apache I do not have issues using same server certificates.

Configuration is using recent (August) mostly stock Caddy with http.proxy running under Windows 7. I control the complete server and client application sides of the connection. I don’t use transparent on the proxy as it messes up forwarding on localhost to apps by replacing ip address with hostname in the request. Anyway that is probably not relevant since it doesn’t seem to be used.

The server application on other end of proxy doesn’t matter as I can verify that it is not using the internal application port (using rawcap, netstat, tcpview) so the 403 is not from the application. I’ve tried to use wireshark but it uses elliptic curve encryption which defeats that so then tried fiddler but there is nothing obvious and they just show the failure response. Wireshark on the server side doesn’t show much but traffic from working and failed clients seem mostly the same so doesn’t look like different certificates are being used and both are selecting same EC encryption method.

The CA is internal self-signed and clients have client key from same ca (though no way to force caddy to require that yet). When I use curl and openssl I can try to use those but doesn’t matter. Also problem occurs the same on all browsers, postman, curl, openssl s_client so I tend to believe this is caddy or go serving up the error but I cannot tell why.

Most machines that I test with can call this endpoint and get the simple response without issue as expected. But I have one machine that cannot complete the call. I don’t have control over the machine and I suspect there may be ssl mitm hardware or software around the machine’s network but hard to say as IT will not respond to inquiries. Not that it matters as I would still like to understand how to troubleshoot this kind of issue.

Access log shows something like the following where the first statement is the failure from bad client and second is from my dev machine. Error log shows nothing.

xxx.xxx.xxx.xxx - - [25/Sep/2017:15:24:18 +0000] "GET /path/test HTTP/1.1" 403 38
xxx.xxx.xxx.xxx - - [25/Sep/2017:15:24:21 +0000] "GET /path/test HTTP/1.1" 200 34

I tried the errors visible setting but doesn’t do anything from what I can see.

So is there any better way to troubleshoot this kind of problem?

The config file looks approximately like the following with path and ports changed:

:443
gzip
log logs/access.log {
	rotate_size 10  # Rotate after 10 MB
	rotate_age  90  # Keep rotated files for 90 days
	rotate_keep 20  # Keep at most 20 log files
	rotate_compress # Compress rotated log files in gzip format
}
errors logs/error.log {
	rotate_size 10  # Rotate after 10 MB
	rotate_age  90  # Keep rotated files for 90 days
	rotate_keep 20  # Keep at most 20 log files
	rotate_compress # Compress rotated log files in gzip format
}
tls conf/ssl/server.crt conf/ssl/server.key {
    ca conf/ssl/caroot.pem
}
proxy /path 127.0.0.1:12345 {
    header_upstream X-Real-IP {remote}
    header_upstream X-Forwarded-Host {host}
    header_upstream X-Forwarded-For {remote}
    header_upstream X-Forwarded-Proto {scheme}
}

(Matthew Fay) #2

This is by design, and certainly will confuse a back end application if it expects some other value for Host. The transparent preset serves a specific use case that may be adverse to your actual requirements. I would only use it with the understanding that my back end application is expecting the set of headers that it configures.

Are you able to share your working Apache configuration for this site?

Also, is that your entire current Caddyfile? As far as I know, none of the directives you’ve configured will produce a 403 for a directory starting with /path unless the upstream server returns one. Timing out or being unable to reach the upstream server will result in a 5xx error instead.


(Theo) #3

Thanks. I sort of figured transparent behaved that way thus the changes I have.

Dang. Figured out the problem. In trying to not to expose internal company network details I left out the ipfilter plugin which was in an included file. That plugin seems to have no logging on block so that appears to be the main part of my problem. The machine was apparently moved to a different network so that software filter blocked it.


(system) #4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.