Caddy v2 behind caddy v2 RevProxy serves empty page

Help
1

1. Caddy version: v2.2.1

2. How I run Caddy:

a. System environment:

2x Ubuntu 20.04 LTS (1x RevProxy, 1x Webserver)

c. Service/unit/compose file:

[Unit]
Description=Caddy
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

RevProxy (192.168.18.1):

daddlemania.de www.daddlemania.de {
  reverse_proxy https://192.168.15.19:443
  tls redacted@redacted.de
  log {
    level warn
  }
  encode zstd gzip
}

Webserver:

192.168.15.19:443 {
    root * /var/www/daddlemania.de
    file_server {
        index index.html
    }
    handle_errors {
        rewrite * /error.html
        file_server
    }
    tls /etc/caddy/ssl/chlin15-19.crt /etc/caddy/ssl/chlin15-19.key
    encode zstd gzip
    log {
    level warn
  }
}

3. The problem I’m having:

When accessing the webpage under its FQDN (https://daddlemania.de), it only serves a blank page.
Still, I’m getting a HTTP 200 response.

4. Error messages and/or full log output:

RevProxy:

Summary 
Nov 25 13:47:57 chlin18-1 caddy[36247]:
{
	"level": "info",
	"ts": 1606308477.2009142,
	"logger": "http.log.access.log15",
	"msg": "handled request",
	"request": {
		"remote_addr": "192.168.18.254:12437",
		"proto": "HTTP/2.0",
		"method": "GET",
		"host": "daddlemania.de",
		"uri": "/",
		"headers": {
			"Sec-Fetch-User": [
				"?1"
			],
			"Sec-Fetch-Dest": [
				"document"
			],
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"
			],
			"Accept": [
				"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
			],
			"Sec-Fetch-Site": [
				"none"
			],
			"Sec-Fetch-Mode": [
				"navigate"
			],
			"Upgrade-Insecure-Requests": [
				"1"
			],
			"Accept-Encoding": [
				"gzip, deflate, br"
			],
			"Accept-Language": [
				"de-DE,de;q=0.9"
			]
		},
		"tls": {
			"resumed": false,
			"version": 772,
			"cipher_suite": 4865,
			"proto": "h2",
			"proto_mutual": true,
			"server_name": "daddlemania.de"
		}
	},
	"common_log": "192.168.18.254 - - [25/Nov/2020:13:47:57 +0100] \"GET / HTTP/2.0\" 200 0",
	"duration": 0.002416423,
	"size": 0,
	"status": 200,
	"resp_headers": {
		"Server": [
			"Caddy",
			"Caddy"
		],
		"Content-Length": [
			"0"
		],
		"Date": [
			"Wed, 25 Nov 2020 12:47:57 GMT"
		]
	}
}

Nov 25 13:47:57 chlin18-1 caddy[36247]:
{
	"level": "info",
	"ts": 1606308477.3852676,
	"logger": "http.log.access.log15",
	"msg": "handled request",
	"request": {
		"remote_addr": "192.168.18.254:12437",
		"proto": "HTTP/2.0",
		"method": "GET",
		"host": "daddlemania.de",
		"uri": "/favicon.ico",
		"headers": {
			"Accept-Encoding": [
				"gzip, deflate, br"
			],
			"Accept-Language": [
				"de-DE,de;q=0.9"
			],
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"
			],
			"Accept": [
				"image/avif,image/webp,image/apng,image/*,*/*;q=0.8"
			],
			"Sec-Fetch-Site": [
				"same-origin"
			],
			"Sec-Fetch-Mode": [
				"no-cors"
			],
			"Sec-Fetch-Dest": [
				"image"
			],
			"Referer": [
				"https://daddlemania.de/"
			]
		},
		"tls": {
			"resumed": false,
			"version": 772,
			"cipher_suite": 4865,
			"proto": "h2",
			"proto_mutual": true,
			"server_name": "daddlemania.de"
		}
	},
	"common_log": "192.168.18.254 - - [25/Nov/2020:13:47:57 +0100] \"GET /favicon.ico HTTP/2.0\" 200 0",
	"duration": 0.003180022,
	"size": 0,
	"status": 200,
	"resp_headers": {
		"Server": [
			"Caddy",
			"Caddy"
		],
		"Content-Length": [
			"0"
		],
		"Date": [
			"Wed, 25 Nov 2020 12:47:57 GMT"
		]
	}
}

Webserver:

Summary 
Nov 25 13:47:57 chlin15-19 caddy[35052]:
{
	"level": "info",
	"ts": 1606308477.198816,
	"logger": "http.log.access",
	"msg": "handled request",
	"request": {
		"remote_addr": "192.168.18.1:59514",
		"proto": "HTTP/2.0",
		"method": "GET",
		"host": "daddlemania.de",
		"uri": "/",
		"headers": {
			"X-Forwarded-Proto": [
				"https"
			],
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"
			],
			"Sec-Fetch-Site": [
				"none"
			],
			"Sec-Fetch-Mode": [
				"navigate"
			],
			"Sec-Fetch-User": [
				"?1"
			],
			"Upgrade-Insecure-Requests": [
				"1"
			],
			"Accept-Encoding": [
				"gzip, deflate, br"
			],
			"Accept-Language": [
				"de-DE,de;q=0.9"
			],
			"X-Forwarded-For": [
				"192.168.18.254"
			],
			"Accept": [
				"text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"
			],
			"Sec-Fetch-Dest": [
				"document"
			]
		},
		"tls": {
			"resumed": false,
			"version": 772,
			"cipher_suite": 4865,
			"proto": "h2",
			"proto_mutual": true,
			"server_name": ""
		}
	},
	"common_log": "192.168.18.1 - - [25/Nov/2020:13:47:57 +0100] \"GET / HTTP/2.0\" 0 0",
	"duration": 0.000008446,
	"size": 0,
	"status": 0,
	"resp_headers": {
		"Server": [
			"Caddy"
		]
	}
}

Nov 25 13:47:57 chlin15-19 caddy[35052]:
{
	"level": "info",
	"ts": 1606308477.382703,
	"logger": "http.log.access",
	"msg": "handled request",
	"request": {
		"remote_addr": "192.168.18.1:59514",
		"proto": "HTTP/2.0",
		"method": "GET",
		"host": "daddlemania.de",
		"uri": "/favicon.ico",
		"headers": {
			"User-Agent": [
				"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.198 Safari/537.36"
			],
			"Sec-Fetch-Site": [
				"same-origin"
			],
			"Sec-Fetch-Mode": [
				"no-cors"
			],
			"Accept-Encoding": [
				"gzip, deflate, br"
			],
			"Accept": [
				"image/avif,image/webp,image/apng,image/*,*/*;q=0.8"
			],
			"Sec-Fetch-Dest": [
				"image"
			],
			"Accept-Language": [
				"de-DE,de;q=0.9"
			],
			"X-Forwarded-For": [
				"192.168.18.254"
			],
			"Referer": [
				"https://daddlemania.de/"
			],
			"X-Forwarded-Proto": [
				"https"
			]
		},
		"tls": {
			"resumed": false,
			"version": 772,
			"cipher_suite": 4865,
			"proto": "h2",
			"proto_mutual": true,
			"server_name": ""
		}
	},
	"common_log": "192.168.18.1 - - [25/Nov/2020:13:47:57 +0100] \"GET /favicon.ico HTTP/2.0\" 0 0",
	"duration": 0.000008855,
	"size": 0,
	"status": 0,
	"resp_headers": {
		"Server": [
			"Caddy"
		]
	}
}

5. What I already tried:

Tried changing the internal connection from HTTPS to HTTP, didn’t help.
Accessing the webserver from the local network directly works fine, curling it from the reverse proxy works perfectly aswell - just everything going through the reverse proxy is’nt.

The same reverse proxy with the same config still serves a lot of sites on Caddy v1, they’re working flawlessly. For some reason, this only happens when the webserver is a v2.

2

Thanks for all the details. That makes it pretty easy. I think the problem is the Host header.

In the reverse_proxy documentation:

By default, Caddy passes thru incoming headers to the backend—including the Host header

So the backend is getting requests for Host: daddlemania.de but is only configured to serve requests for Host: 192.168.15.19.

The docs show how to change the Host header to whatever you want/need.

1 Like
3

It might work to change 192.168.15.19:443 to simply :443 so that it doesn’t try to match a specific Host header (on the Caddy instance that is the file server)

1 Like
4

Hey there, thanks to you both! It works flawlessly now.

After confirming that it was caused by the host-header by removing the IP the webserver binds to as @francislavoie said, I changed the reverse proxy directive so that the header is being modified to match the webserver’s IP:

reverse_proxy https://192.168.15.19 {
    header_up Host {http.reverse_proxy.upstream.hostport}
  }

After thinking about that again for a bit and deciding that it isn’t that great of a solution when running multiple webpages on one system, I figured that, since I wouldn’t be using the server locally anyways, I could just let caddy match the FQDN on the webserver aswell. Since I’m providing the certificates myself, it wouldn’t try to get them from Let’s Encrypt.
Another benefit of this is, that the webserver can behave different based on which Domain has been used to access him.

And that’s where I am right now! Thanks for the help and lesson, I learned quite a bit from it :slight_smile:

1 Like
5

This topic was automatically closed after 30 days. New replies are no longer allowed.