Redirection issue with multiple chained Caddy reverse proxy servers

[DrDoug88]

1. Output of caddy version: v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=

2. How I run Caddy:

Two caddy servers primarily for reverse-proxying. One server is internet facing, and the other is on internal vpn.

a. System environment:

Linux Ubuntu, Docker

b. Command:

Docker-based

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

External Internet-Reachable Caddyfile:

https://service1.externalinternet.com {
  reverse_proxy https://service1.vpn {
    header_up Host {http.reverse_proxy.upstream.host}
  }
}

https://service2.externalinternet.com {
  reverse_proxy https://service2.vpn {
    header_up Host {http.reverse_proxy.upstream.host}
  }
}

https://service3.externalinternet.com {
  reverse_proxy https://service3.vpn {
    header_up Host {http.reverse_proxy.upstream.host}
  }
}

https://service4.externalinternet.com {
  reverse_proxy https://service4.vpn {
    header_up Host {http.reverse_proxy.upstream.host}
  }
}

Internal VPN Caddyfile:

https://service1.vpn {
  reverse_proxy https://docker1 {
    trusted_proxies private_ranges
  }
}

https://service2.vpn {
  reverse_proxy https://docker2 {
    trusted_proxies private_ranges
  }
}

https://service3.vpn {
  reverse_proxy https://docker3 {
    trusted_proxies private_ranges
  }
}

https://service4.vpn {
  reverse_proxy https://docker4 {
    trusted_proxies private_ranges
  }
}

3. The problem I’m having:

I am running 2 caddy servers for reverse proxying. One server is reachable from the external internet, and the other is on an internal vpn only accessible from vpn clients. I have 4 docker-based services running behind the internal vpn caddy server. DNS and TLS all work fine and correctly. This configuration is mostly derived from this article (Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts)

5. What I already tried:

Based on this setup, everything is working correctly except for one service, service4. All 4 services require a login process. However, when accessing service4 from the outer Caddy instance, i.e. https://service4.externalinternet.com, it will redirect to https://service4.vpn after logging in. If I reload https://service4.externalinternet.com, then I am logged into service4 with external hostname (https://service4.externalinternet.com), and I can proceed to navigate the site. The same behavior occurs when logging out of the site. It will redirect to https://service4.vpn rather than https://service4.externalinternet.com.

If I add “header_up Host {header.X-Forwarded-Host}” to the reverse_proxy block of service4 on the backend Caddy server, then everything works correctly when accessing via the external internet hostname - logging in/out will redirect to external internet-based hostname (https://service4.externalinternet.com). However, when accessing service4 via the vpn, i.e. https://service4.vpn, it will redirect to https://docker4 rather than https://service4.vpn, when logging in/out. I can think of a few contrived ways to handle this specific situation for service4, but is there a clean and correct way to handle this? I apologize for my ignorance on this matter, this is not an area of software engineering that I am entirely familiar with.

Thank you for all of the hard work put into Caddy, and thank you in advance for your time on this matter.

6. Links to relevant resources:

Use Caddy for local HTTPS (TLS) between front-end reverse proxy and LAN hosts

TL;DL This Wiki explains how to enable https connections between hosts in a LAN with automatically renewed certificates. Just scroll down and go over the example Caddyfiles. They hopefully have enough comment to understand the idea Introduction If you have successfully followed the Wiki Using Caddy as a reverse proxy in a home network by @Matt, you have setup a reverse proxy that provides a TLS encrypted connection from the internet to that reverse proxy. This is mandatory and you shou…

[francislavoie]

@DrDoug88

1. Output of caddy version: v2.5.1 h1:bAWwslD1jNeCzDa+jDCNwb8M3UJ2tPa8UZFFzPVmGKs=

Please upgrade to v2.5.2

@DrDoug88

header_up Host {http.reverse_proxy.upstream.host}

FYI, you can shorten this to:

header_up Host {upstream_hostport}

@DrDoug88

If I add “header_up Host {header.X-Forwarded-Host}” to the reverse_proxy block of service4 on the backend Caddy server, then everything works correctly when accessing via the external internet hostname

Ultimately, this is an issue with your upstream app. Caddy will send through the X-Forwarded-Host header with the correct hostname (as I think you know). Your upstream app should read from that (ideally, if configured with trusted proxies itself, but not all applications do that correctly). I suggest you open an issue with the devs of that app to get them to fix this.

That said, somewhat contrived, but you could do this on your internal Caddy instance

(host-var) {
	@isExternal header X-Forwarded-Host *
	vars @isExternal actualHost {header.X-Forwarded-Host}
	vars actualHost {host}
}

https://service4.vpn {
	import host-var
	reverse_proxy https://docker4 {
		header_up Host {vars.actualHost}
		trusted_proxies private_ranges
	}
}

Basically, constructs a variable depending on whether the incoming request had the X-Forwarded-Host header or not (it shouldn’t for directly internal connections, unless the client is trying to spoof the header, but why would someone connected to your VPN do that anyways?) and then you can use that variable in the proxy.

[DrDoug88]

Thank you @francislavoie. This solved the issue. I had to stare at the (host-var) snippet for a while to understand what is going on, but I think I do now. Correct me if I am wrong, but here is what I gather:

@isExternal is a named Request Matcher that, when the X-Forwarded-Host contains anything, i.e. defined by the wildcard (*), then it will match on “vars @isExternal actualHost {header.X-Forwarded-Host}”, thus setting actualHost equal to header.X-Forwarded-Host, covering the case when the request is coming in through outer proxy. If the request is internal, then X-Forwarded-Host will not be set, thus setting actualHost equal to host. The snippet is imported into the reverse proxy block and header_up Host is set to what was just matched and set in vars.actualHost.

Is this correct? I’m sorry, just trying to get a better understanding of the syntax based on what I can find in the documentation.

As far as the upstream service4 app being the culprit, this is what I had suspected, since I wasn’t experiencing issues with the other apps.

On an internal vpn, one shouldn’t suspect headers to be spoofed, but I’m approaching this as if everything is adversarial. In this case, service4 shouldn’t normally be exposed to public internet, but I’m testing all possible cases to see if there are any issues.

Thank you kindly for your prompt response.

[francislavoie]

@DrDoug88

Is this correct? I’m sorry, just trying to get a better understanding of the syntax based on what I can find in the documentation.

Yep! That’s right!