I am trying something a bit weird. I have 2 systems (1 and 2). System 1 uses cloudflare to generate valid certs for my internal LAN (dl.l.bradford.la and tv.l.bradford.la). tv.l.bradford.la proxies great, no issues.
dl.l.bradford.la has issues acting as a reverse proxy to another caddy reverse proxy (system 2). This second reverse_proxy will ideally act as a reverse proxy for several docker services as subdirectories of dl.l.bradford.la. It doesn’t seem to work properly, getting hung up on tls. I added the transport httptls_insecure_skip_verify to hopefully help that, but it still is giving an error 502.
4. Error messages and/or full log output:
Navigating to https://dl.l.bradford.la/nzbget/ gives the error message in caddy on system 1 (no log output on system 2)
Google-fu and refining my caddy files to try and eliminate any other possibility. It seems it is finally hung up on the tls connection between caddy 1 and caddy 2.
Don’t use caddy/caddy. Use caddy instead. The caddy/caddy image is our CI target, and it should not be used outside of our internal use.
Path matching is exact in Caddy. That will only handle exactly /nzbget/ and nothing else. You should use /nzbget/* instead.
I’d recommend using subdomains instead of subpaths. See this article:
Turn on the debug global option on both servers, to see more detailed logs.
I think what’s happening is SNI gets set to 172.16.1.50 instead of dl.l.bradford.la which means the upstream doesn’t know the correct certificate to choose.
Why are you proxying over HTTPS? If this is in your local network, then there’s probably no problem with proxying over HTTP instead. The only concern is if there’s untrusted devices in your network that could intercept the traffic. But that’s probably unlikely.
When I change system 1 config to remove the https and port 443 bit, and system 2 to remove the 443 port on the service block, I get different errors, tls: first record does not look like a TLS handshake, as if system 1 is trying to force a ssl connection anyway:
Finally, I commented out the transport http block in system 1’s Caddyfile, and when I tried to access https://dl.l.bradford.la/nzbget/ I get a flood of these messages in system 1 (with the selected upstream and upstream roundtrip messages repeating forever, seems the browser keeps retrying since it’s a 308?):
The important part is that the incoming connection to Caddy is encrypted, over the public internet. Once it’s inside of your network, then the risk is drastically lower.
This looks wrong. It seems like you had the proxy configured to do a TLS but to port 80, which is mismatched.
If you make an HTTP request to a server that’s expecting HTTPS, then it’ll serve an HTTP->HTTPS redirect. So you’d have to change your 2nd server to only listen on port 80 with :80 or prefix the domain with http:// (depending on your routing needs).
That seems to have fixed it, I’m able to access my services in subdirectories. Thanks!
I still want to force TLS between service 1 and service 2. I thought it would just take uncommenting the tls_insecure_skip_verify and setting the destination to https in system 1, while changing :80 to :443 in system 2, but I get 502 errors again, such as these on system 2:
"no matching certificates and no custom selection logic"
"all external certificate managers yielded no certificates and no errors"
"no certificate matching TLS ClientHello"
"http: TLS handshake error from 172.16.1.71:60680: no certificate available for '172.18.0.6'"
172.18.0.6 is the internal docker IP for system 2 caddy. Looks like it’s not automatically generating the certs like I would expect tls internal to do on system 2.
The tricky thing to get right is SNI and host matching. If those aren’t right, then the TLS connection between them will still fail.
Using tls_insecure_skip_verify means you’re turning off all security so it’s effectively the same as using HTTP. The data’s encrypted, but it can be trivially man-in-the-middle attacked. And using TLS adds overhead (increased latency) so there’s essentially no benefit to it unless you do establish trust. And that involves copying the root CA cert from the upstream server to your front one and pointing the transport config to trust certificates signed by that root.
Looking closer at the logs, I’m seeing that "server_name":"" and "sni":"". So it doesn’t look like any SNI is being passed from system 1, which is starting to make sense.
Reading this got me to tls_server_name which got me to explicitly pass on the SNI to system 2. All good! TLS proxying between system 1 and system 2 is working. Now on to getting proper certs.
Is there any way to use the same Lets Encrypt cert (that system 1 negotiates automatically) for the connection between system 1 and 2, since the SNI is the same? Any way to share these certs between caddy instances that’s in-built?
If not, I will probably use tls_dns_cloudflare on server 2 to simplify valid cert management.
Ultimately, I moved from subdirectories (it was a PITA to get each service to properly behave) to subdomains. I use a wildcard subdomain site block and the cloudflare dns provider works great and no more reverse proxying to another reverse proxy. Each system takes care of its own certs.