Error 502 with Cloudflare

Angius · March 11, 2022, 1:37pm

1. Caddy version (`caddy version`):

v2.4.6 h1:HGkGICFGvyrodcqOOclHKfvJC0qTU7vny/7FhYp9hNw=

2. How I run Caddy:

Pure, clean installation, except with the Cloudflare module added via add-package command.

ASP.NET API running at localhost:500 (http) and localhost:5001 (https)

a. System environment:

Debian 10

b. Command:

caddy run

or

caddy start

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

images.genfic.net {
        tls {
                dns cloudflare [Cloudflare Token]
        }
        reverse_proxy 127.0.0.1:5001 {
                header_up Host {upstream_hostport}
                header_up X-Forwarded-Host {host}
        }
}

3. The problem I’m having:

I’m trying to reverse-proxy my subdomain to an API running on my VPS. The domain is already routed through Cloudflare, since I need its edge caching.

Unfortunately, even after a couple of hours fiddling with various random docs and tutorials, installing Go and what not, I can’t get it to work. Visiting the address shows error 502, and the same error shows up in Caddy’s logs.

4. Error messages and/or full log output:

$ sudo caddy run debug
2022/03/11 13:33:38.395 INFO    using adjacent Caddyfile
2022/03/11 13:33:38.403 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/03/11 13:33:38.404 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS   {"server_name": "srv0", "https_port": 443}
2022/03/11 13:33:38.405 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/03/11 13:33:38.406 INFO    http    enabling automatic TLS certificate management   {"domains": ["images.genfic.net"]}
2022/03/11 13:33:38.408 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/03/11 13:33:38.408 INFO    serving initial configuration
2022/03/11 13:33:38.409 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc000572e00"}
2022/03/11 13:33:38.409 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/03/11 13:33:38.411 INFO    tls     finished cleaning storage units
2022/03/11 13:33:51.757 ERROR   http.log.error  EOF     {"request": {"remote_addr": "141.101.77.227:21808", "proto": "HTTP/2.0", "method": "GET", "host": "images.genfic.net", "uri": "/", "headers": {"Accept-Encoding": ["gzip"], "Cf-Ipcountry": ["FR"], "Cf-Ray": ["6ea4aa0defc097ac-AMS"], "X-Forwarded-Proto": ["https"], "Cf-Connecting-Ip": ["146.59.95.130"], "Cdn-Loop": ["cloudflare"], "X-Forwarded-For": ["146.59.95.130"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "User-Agent": ["curl/7.64.0"], "Accept": ["*/*"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "images.genfic.net"}}, "duration": 0.012296872, "status": 502, "err_id": "r1a1mk4nm", "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"}

5. What I already tried:

I tried using Xcaddy instead, but to no avail

6. Links to relevant resources:

emilylange · March 11, 2022, 3:44pm

Hi

Have you tried adding https:// to your reverse_proxy upstream, which is required for https upstreams that don’t use :443 (in your case :5001)?

images.genfic.net {
        tls {
                dns cloudflare [Cloudflare Token]
        }
-        reverse_proxy 127.0.0.1:5001 {
+        reverse_proxy https://127.0.0.1:5001 {
                header_up Host {upstream_hostport}
                header_up X-Forwarded-Host {host}
        }
}

See caddyfile/directives/reverse_proxy#upstreams and the example in caddyfile/directives/reverse_proxy#https

Angius · March 11, 2022, 4:09pm

Thanks for the answer! Just did that, and unfortunately no cigar:

$ sudo caddy run
2022/03/11 16:08:14.198 INFO    using adjacent Caddyfile
2022/03/11 16:08:14.204 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/03/11 16:08:14.205 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2022/03/11 16:08:14.206 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/03/11 16:08:14.207 INFO    http    enabling automatic TLS certificate management   {"domains": ["images.genfic.net"]}
2022/03/11 16:08:14.211 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/03/11 16:08:14.211 INFO    serving initial configuration
2022/03/11 16:08:14.212 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00028b110"}
2022/03/11 16:08:14.212 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/03/11 16:08:14.217 INFO    tls     finished cleaning storage units
2022/03/11 16:08:28.245 ERROR   http.log.error  dial tcp 127.0.0.1:5001: connect: connection refused    {"request": {"remote_addr": "162.158.203.69:59780", "proto": "HTTP/2.0", "method": "GET", "host": "images.genfic.net", "uri": "/", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0"], "Accept-Language": ["en-GB,en;q=0.5"], "Dnt": ["1"], "Cdn-Loop": ["cloudflare"], "X-Forwarded-For": ["178.43.215.78"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Cf-Ipcountry": ["PL"], "Sec-Fetch-User": ["?1"], "Sec-Gpc": ["1"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Mode": ["navigate"], "Cf-Connecting-Ip": ["178.43.215.78"], "Cf-Ray": ["6ea58c87ce854174-HAM"], "X-Forwarded-Proto": ["https"], "Accept-Encoding": ["gzip"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"], "Sec-Fetch-Site": ["none"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "images.genfic.net"}}, "duration": 0.001221238, "status": 502, "err_id": "pbit64mrz", "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"}

Edit: I forgot to start the actual app it’s supposed to be proxying to… But now I did, and there seems to be no change:

$ sudo caddy run
2022/03/11 16:10:59.946 INFO    using adjacent Caddyfile
2022/03/11 16:10:59.954 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2022/03/11 16:10:59.956 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2022/03/11 16:10:59.956 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2022/03/11 16:10:59.959 INFO    http    enabling automatic TLS certificate management   {"domains": ["images.genfic.net"]}
2022/03/11 16:10:59.961 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2022/03/11 16:10:59.962 INFO    serving initial configuration
2022/03/11 16:10:59.963 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2022/03/11 16:10:59.965 INFO    tls     finished cleaning storage units
2022/03/11 16:10:59.966 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0001dab60"}
2022/03/11 16:11:03.709 ERROR   http.log.error  x509: cannot validate certificate for 127.0.0.1 because it doesn't contain any IP SANs  {"request": {"remote_addr": "162.158.203.69:61128", "proto": "HTTP/2.0", "method": "GET", "host": "images.genfic.net", "uri": "/", "headers": {"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:99.0) Gecko/20100101 Firefox/99.0"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"], "Sec-Gpc": ["1"], "Cdn-Loop": ["cloudflare"], "Accept-Encoding": ["gzip"], "X-Forwarded-Proto": ["https"], "Sec-Fetch-Mode": ["navigate"], "Cf-Ipcountry": ["PL"], "X-Forwarded-For": ["178.43.215.78"], "Dnt": ["1"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-Site": ["none"], "Sec-Fetch-User": ["?1"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Accept-Language": ["en-GB,en;q=0.5"], "Cf-Connecting-Ip": ["178.43.215.78"], "Cf-Ray": ["6ea590519cb74174-HAM"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "proto_mutual": true, "server_name": "images.genfic.net"}}, "duration": 0.263603043, "status": 502, "err_id": "qgd44yv8w", "err_trace": "reverseproxy.statusError (reverseproxy.go:886)"}

emilylange · March 11, 2022, 5:30pm

That’s a different error now - partial success!

Caddy won’t trust self-signed certificates by default.

You have two main possibilities now:

Undermine any security you get from running encrypted http for reverse_proxy traffic between caddy and your service by completely disabling certificate verification, like

images.genfic.net {
    tls {
        dns cloudflare [Cloudflare Token]
    }
    reverse_proxy https://127.0.0.1:5001 {
        header_up Host {upstream_hostport}
        header_up X-Forwarded-Host {host}
+       transport http {
+           tls
+           tls_insecure_skip_verify
+       }
    }
}

For more options, check caddyfile/directives/reverse_proxy#the-http-transport.

Import your service’s certificate into your host’s trust chain. This will differ from service to service (and OS).

matt · March 11, 2022, 5:44pm

More precisely, the backend is presenting a certificate that does not contain 127.0.0.1 as a subject name. So even if it’s a trusted cert, it won’t verify with Caddy because the subject doesn’t match.

If the backend’s certificate is already trusted but contains a domain name, then you need to set tls_server_name in the HTTP transport.

Btw, if method 1 is followed (tls_insecure_skip_verify), the upstream address can drop the https:// since HTTPS is being explicitly enabled.

The app not running would definitely explain EOF. And there is a change, as @IndeedNotJames said. Notice the error is different.

Angius · March 11, 2022, 6:09pm

I hoped the Cloudflare plugin is supposed to handle all of that, and that’s why it needed an API key?

Far as disabling the verification between Caddy and my service, how much of a security risk is it, seeing how they’re both running on the same VPS?

And regarding the second option, I assume I’d need to get the Let’s Encrypt cert that Caddy generated… somehow, and give it to Cloudflare?

Or perhaps I could disable Cloudflare’s SSL… somehow and only rely on Caddy for that? Not sure how to do it either, though.

CC @matt

matt · March 11, 2022, 6:51pm

The cloudflare plugin gives Caddy a certificate for domains’ nameservers at Cloudflare, assuming images.genfic.net has its NS there. It doesn’t have anything to do with your reverse proxy.

emilylange · March 11, 2022, 8:01pm

I just wanted to say that there is basically no benefit in using https:// with disabled certificate verification instead of plain http in your use-case.

But I would argue you should be fine taking the first option, disabling certificate verification

For the second option, you would have to look up how your service, not Caddy itself (in your case, some asp.net stack) handles certificates generation and whatnot.

Browser → LetsEncrypt (or ZeroSSL) → Caddy → reverse_proxy (http or self-signed https) → Your service upstream

Changing Cloudflare’s SSL has no influence on how Caddy handles reverse_proxy traffic between Caddy and the service upstream.

system · April 10, 2022, 1:38pm

This topic was automatically closed after 30 days. New replies are no longer allowed.