Caddy Reverse Proxy to Multiple Sites on Port 80

Help
1

1. Output of caddy version:

v2.6.1 h1:EDqo59TyYWhXQnfde93Mmv4FJfYe00dO60zMiEt+pzo=

2. How I run Caddy:

a. System environment:

Ubuntu 22, fresh install, all updates. ONLY caddy installed on top (so far)

b. Command:

caddy reload --config /home/caddy/Caddyfile

c. Service/unit/compose file:

Nope/nada

d. My complete Caddy config:

sp4.matice.com {
        encode zstd gzip
        respond "Hello!"
}
matice.com {
        reverse_proxy https://216.108.230.119
}
myseconddomain.com {
        reverse_proxy https://216.108.230.119
}
mythirddomain.com {
        reverse_proxy https://216.108.230.119
}

3. The problem I’m having:

Objective is to have caddy sit in front of an existing webserver that already runs multiple sites, all on port 80/443 (80 redirects to 443 if necessary on that server). That server is running AAPanel to serve sites. Caddy loads fine, and the “http or https://sp4.matice.com” test works fine. Going to matice.com brings back a blank page. Same with any other domain I move to this IP and test. I have also tested using an internal ip address, forcing the :443 on the IP, removing https:// from front.

4. Error messages and/or full log output:


Sep 25 10:24:31 ashburn-va-datacenter.serverpoint.com caddy[50307]: {"level":"debug","ts":1664115871.5614827,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/deBFzjS1CRmJNaValUHMEQ","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["451"],"Content-Type":["application/json"],"Date":["Sun, 25 Sep 2022 14:24:31 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["5nmHi-2p99GLVWsdqQ4L8owGosl2DB7MO9sLvZajW_w"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 25 10:24:32 ashburn-va-datacenter.serverpoint.com caddy[50307]: {"level":"debug","ts":1664115872.4457216,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme.zerossl.com/v2/DV90/authz/smPwtDEq6427eEl13zbUaA","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.6.1 CertMagic acmez (linux; amd64)"]},"response_headers":{"Access-Control-Allow-Origin":["*"],"Cache-Control":["max-age=-1"],"Content-Length":["450"],"Content-Type":["application/json"],"Date":["Sun, 25 Sep 2022 14:24:32 GMT"],"Link":["<https://acme.zerossl.com/v2/DV90>;rel=\"index\""],"Replay-Nonce":["cyk491HRipR_StmSoV3LwG11JMZOTbEd5ILnQRNRlI0"],"Retry-After":["5"],"Server":["nginx"],"Strict-Transport-Security":["max-age=15552000"]},"status_code":200}
Sep 25 10:24:33 ashburn-va-datacenter.serverpoint.com caddy[50307]: {"level":"debug","ts":1664115873.2876794,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"216.108.230.199:443","duration":3.000317322,"request":{"remote_ip":"73.180.158.177","remote_port":"59635","proto":"HTTP/2.0","method":"GET","host":"matice.com","uri":"/","headers":{"Accept-Language":["en-US,en;q=0.5"],"X-Forwarded-Proto":["https"],"Te":["trailers"],"Sec-Fetch-Site":["none"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Upgrade-Insecure-Requests":["1"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["73.180.158.177"],"Sec-Fetch-User":["?1"],"Cookie":[],"X-Forwarded-Host":["matice.com"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0"],"Sec-Fetch-Dest":["document"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matice.com"}},"error":"dial tcp 216.108.230.199:443: i/o timeout"}
Sep 25 10:24:33 ashburn-va-datacenter.serverpoint.com caddy[50307]: {"level":"error","ts":1664115873.288417,"logger":"http.log.error","msg":"dial tcp 216.108.230.199:443: i/o timeout","request":{"remote_ip":"73.180.158.177","remote_port":"59635","proto":"HTTP/2.0","method":"GET","host":"matice.com","uri":"/","headers":{"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Cookie":[],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-Mode":["navigate"],"Te":["trailers"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matice.com"}},"duration":3.001553079,"status":502,"err_id":"ihqhvu7d3","err_trace":"reverseproxy.statusError (reverseproxy.go:1271)"}
Sep 25 10:24:33 ashburn-va-datacenter.serverpoint.com caddy[50307]: {"level":"debug","ts":1664115873.6374648,"logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"216.108.230.199:443","total_upstreams":1}
Sep 25 10:24:33 ashburn-va-datacenter.serverpoint.com caddy[50307]: {"level":"debug","ts":1664115873.6375327,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"216.108.230.199:443","duration":0.000012519,"request":{"remote_ip":"73.180.158.177","remote_port":"59635","proto":"HTTP/2.0","method":"GET","host":"matice.com","uri":"/favicon.ico","headers":{"Referer":["https://matice.com/"],"Te":["trailers"],"Accept":["image/avif,image/webp,*/*"],"Accept-Language":["en-US,en;q=0.5"],"Cookie":[],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:106.0) Gecko/20100101 Firefox/106.0"],"X-Forwarded-Proto":["https"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["image"],"X-Forwarded-For":["73.180.158.177"],"X-Forwarded-Host":["matice.com"]},"tls":{"resumed":true,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"matice.com"}},"error":"context canceled"}

5. What I already tried:

Putting :443 at end of IP
Changing IP to internal IP
Putting :443 at end of internal IP
Removing https:// from front of IP
Adding this section after IP:
{
header_up X-Real-IP {remote}
}

6. Links to relevant resources:

I didn’t find any :frowning:

2

Please help me understand this as it makes NO sense. I tried something I thought would be dumb. I put the external IP of the webserver for matice.com in the local hosts file on caddy. I set caddy to go to https://matice.com. And it worked.

Why would it not work with the IP but did work with the domain name using a local hosts domain overwrite? I don’t want to add every domain into my local hosts file. This seems unnecessary. It seems to me that caddy is not sending the header correctly to the webserver.

3

This is, very simply, because a HTTPS certificate is issued for a domain, not for an IP address.

There is (almost) no chance your upstream server can furnish Caddy with a valid certificate for https://216.108.230.119. The easiest fix is to configure Caddy to connect to the domain itself, rather than an IP; there are other workarounds possible if you must connect over HTTPS to an external IP, but which workaround you need depends on where exactly the connection is breaking down (debug logs will help).

3 Likes
4

Whenever you are proxying to a https backend, there are two pitfalls:

  1. The Host header, which gets inherited from the connection/vhost by default
  2. The TLS SNI/server name, which sets the server name in the initial TLS handshakes with the upstream, basically. That’s before any headers (like the Host header) even get sent

Excerpts from reverse_proxy (Caddyfile directive) — Caddy Documentation

When proxying over HTTPS, you may need to override the Host header such that it matches the TLS SNI value, which is used by servers for routing and certificate selection. See the HTTPS section below for more details.

and

Since (most) headers retain their original value when being proxied, it is often necessary to override the Host header with the configured upstream address when proxying to HTTPS, such that the Host header matches the TLS ServerName value. For example:

reverse_proxy https://example.com {
	header_up Host {upstream_hostport}
}

If your config looks like

matice.com {
        reverse_proxy https://216.108.230.119
}

then Caddy keeps the Host header value set to matice.com, but the TLS server name will be set to 216.108.230.119, to which your upstream server says “nope”

Your workaround (putting the IP+hostname into the hosts file and pointing Caddy to the hostname) works, because Caddy will then use matice.com as TLS server name (SN)

Instead of that, you could also just make Caddy that you don’t want to read the SN from the upstream target (in your case IP) and override it:

matice.com {
	reverse_proxy https://216.108.230.119 {
		transport http {
			tls_server_name matice.com
		}
	}
}

Strictly speaking, you could also drop that leading https:// in your upstream target, because

  • tls uses HTTPS with the backend. This will be enabled automatically if you specify backends using the https:// scheme or port :443, or if any of the below tls_* options are configured.

but feel free to keep it for better readability or whatever /shrug

The actual is tls_server_name option is documented here and here

4 Likes
5

Thanks! This worked great!

6

This topic was automatically closed after 30 days. New replies are no longer allowed.