Caddy 502 error when proxing https upstream (istio)

Help
1

1. Caddy version (caddy version):

v2.4.6

2. How I run Caddy:

vaultwarden-proxy.dev.example.de {
        reverse_proxy 10.110.180.110:30443 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
        tls example@example.de {
                dns cloudflare example
        }
        log {
                output file /var/log/caddy/access-vaultwarden-proxy-dev-example-de.log
        }
}

Also tried:
reverse_proxy 10.110.180.110:30443 {
reverse_proxy / 10.110.180.110:30443 {
reverse_proxy https://10.110.180.110:30443 {

And:
tlswith the verify insecure

a. System environment:

Ubuntu 20.04, systemd

3. What is my problem, error messages and/or full log output:

May 19 15:20:54 gateway-primary-zpx2e caddy[56074]: {"level":"debug","ts":1652973654.9613335,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.110.180.110:30443","duration":0.05179861,"request":{"remote_addr":"147.161.164.112:32251","proto":"HTTP/2.0","method":"GET","host":"vaultwarden-proxy.dev.example.de","uri":"/","headers":{"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"X-Forwarded-Proto":["https"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"101\", \"Google Chrome\";v=\"101\""],"Upgrade-Insecure-Requests":["1"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["147.161.164.112"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"vaultwarden-proxy.dev.example.de"}},"error":"read tcp 10.110.180.157:40986->10.110.180.110:30443: read: connection reset by peer"}

May 19 15:20:54 gateway-primary-zpx2e caddy[56074]: {"level":"error","ts":1652973654.961822,"logger":"http.log.error.log8","msg":"read tcp 10.110.180.157:40986->10.110.180.110:30443: read: connection reset by peer","request":{"remote_addr":"147.161.164.112:32251","proto":"HTTP/2.0","method":"GET","host":"vaultwarden-proxy.dev.example.de","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"Accept-Encoding":["gzip, deflate, br"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"101\", \"Google Chrome\";v=\"101\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"vaultwarden-proxy.dev.example.de"}},"duration":0.05234538,"status":502,"err_id":"p9i6mt9mz","err_trace":"reverseproxy.statusError (reverseproxy.go:886)"}

May 19 15:20:54 gateway-primary-zpx2e caddy[56074]: {"level":"error","ts":1652973654.9620607,"logger":"http.log.access.log8","msg":"handled request","request":{"remote_addr":"147.161.164.112:32251","proto":"HTTP/2.0","method":"GET","host":"vaultwarden-proxy.dev.example.de","uri":"/","headers":{"Accept-Encoding":["gzip, deflate, br"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua":["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"101\", \"Google Chrome\";v=\"101\""],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Upgrade-Insecure-Requests":["1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Sec-Fetch-Site":["none"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Sec-Fetch-Dest":["document"],"Accept-Language":["de-DE,de;q=0.9,en-US;q=0.8,en;q=0.7"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"vaultwarden-proxy.dev.example.de"}},"common_log":"147.161.164.112 - - [19/May/2022:15:20:54 +0000] \"GET / HTTP/2.0\" 502 0","user_id":"","duration":0.05234538,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}

4. What I already tried:

I am trying to use caddy as frontend gateway/loadbalancer for my services running in multiple k8s clusters. Because these are not publicly exposed caddy is running in a VM with a public IP and should reverse proxy these service domains to my cluster services.

This does already work great for some services running in docker and one cluster currently running traefik as ingress (with valid cloudflare/letsencrypt ssl certs). Because of unrelated reasons I want to use istio instead but I cannot get the same proxy working with the istio setup.

vaultwarden.example.de → caddy → traefik → vaultwarden work great, which makes me assume that i just have to redirect to the istio cluster instead of traefik.

But vaultwarden-proxy.dev.example.de returns an 502 error, because of connectio reset (see debug logs), which made me think that istio does not work.

When I set a local dns route to the istio host for vaultwarden-direct.dev.example.de the service is loading fine with valid certificates.

Well and now I am out of ideas whats wrong with my caddy config. The upstream (istio / 10.110.180.110:30443) works, since I am able to use it via a local dns route.

Any ideas would be very much appreciated :slight_smile:

2

Does your upstream app/server have a firewall? Make sure it doesn’t block incoming requests on port 30443.

This doesn’t look like an issue with Caddy, but a networking issue between Caddy and your upstream app.

1 Like
3

No there are some iptables (default), but nothing related to port 30443. When I check open ports I see a LISTENer for port 30443 tcp 0 0 0.0.0.0:30443 0.0.0.0:* LISTEN - . And if I access the server directly (without the caddy in-between) the service loads great.

4

Alright I found something interesting, which leads me to the conclusion that the issue is about SSL.

With this configuration of caddy

vaultwarden-proxy.dev.example.de {
    reverse_proxy 10.110.180.110:30080 {
        transport http {

        }
    }
    tls example@example.de {
            dns cloudflare example
    }
    log {
        output file /var/log/caddy/access-vaultwarden-proxy-dev-example-de.log
    }
}

it works. NOTE: for testing I currently serve http(30080) and https(30443). So http traffic works, but https does not. On the upstream I have a valid letsencrypt SSL cert from cloudflare. I assume that i beside of tls_insecure_skip_verify i need to set something else for caddy to terminate tls and accept the upstream?

5

It depends what your upstream is expecting from Caddy. If it does any TLS-SNI matching, then you might need to play with tls_server_name, etc.

Is this upstream app in a private network? If so then it’s fine to proxy over HTTP, as long as you’re not running untrusted code in your private network.

6

Ahh you are right, the hostname/servername is lost when terminating at caddy. Thanks for raising, this is probably my issue. The connection from caddy to the servers are on a private network, so jea would be okey, but i would like to have it TLS all the way :slight_smile:

Thanks again for pointing me in this direction, I will update this thread when I verified it to work with the server/hostname.

7

This topic was automatically closed after 30 days. New replies are no longer allowed.