Two reverse proxies in a row

Help
1

1. The problem I’m having:

Hi, I have the following setup. I have a network with two machines. Machine 1 is reachable from the internet and is running caddy to reverse proxy to machine 2. Machine 2 is running various docker services. Some are to be reached via internet, some only locally. Until now I have exposed ports for the various services on machine 2 and connected to them from caddy on machine 1. But I want to change this for a cleaner setup and less exposed ports. I would like to have another caddy on machine 2 that then reverse proxies into the docker container that no longer have public ports.
So there are two reverse proxies. Machine 1 is also handling the certificates.
Ideally I would also have https transfer from machine 2 to machine 1 but I couldn’t make that work yet (I have found the other post explaining that).
Now to my problem. On machine 2 my caddyfile looks like this:

http://kimai.lan {
reverse_proxy kimai-nginx:80
}

Which works fine.

On machine 1 my caddyfile says

kimai.domain.tld {
reverse_proxy http://kimai.lan
}

When accessing the external domain I get redirected to the internal domain.
When I add header_up Host {upstream_hostport} it works, at least for Kimai. But not for firefly. It also seems weird to me that I have to use that directive because the documentation says this is only necessary when reverse proxying to another https service, which this isn’t.

When checking the debug window for firefly I see that the webpage is still making requests for the internal .lan domain.

2. Error messages and/or full log output:

Machine 1

DBG | ts=1707833581.8706071 logger=events msg=event name=tls_get_certificate id=e99a93ff-5376-48f5-bdfb-842ef506f6f2 origin=tls data={"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"ff.arianw.de","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"192.168.178.5","Port":50486,"Zone":""},"LocalAddr":{"IP":"172.20.0.4","Port":443,"Zone":""}}} 
DBG | ts=1707833581.8710706 logger=tls.handshake msg=choosing certificate identifier=ff.arianw.de num_choices=1 
DBG | ts=1707833581.8713124 logger=tls.handshake msg=default certificate selection results identifier=ff.arianw.de subjects=["ff.arianw.de"] managed=true issuer_key=acme-v02.api.letsencrypt.org-directory hash=1720b886402f3a012670ef9a40c1d4a1c71a51ca1ad16f37589015959667ba51 
DBG | ts=1707833581.8714604 logger=tls.handshake msg=matched certificate in cache remote_ip=192.168.178.5 remote_port=50486 subjects=["ff.arianw.de"] managed=true expiration=1711277027 hash=1720b886402f3a012670ef9a40c1d4a1c71a51ca1ad16f37589015959667ba51 
DBG | ts=1707833581.9183824 logger=http.handlers.reverse_proxy msg=upstream roundtrip upstream=firefly.lan:80 duration=0.041305609 request={"remote_ip":"192.168.178.5","remote_port":"50486","client_ip":"192.168.178.5","proto":"HTTP/2.0","method":"GET","host":"firefly.lan:80","uri":"/","headers":{"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["192.168.178.5"],"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["ff.arianw.de"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"h2","server_name":"ff.arianw.de"}} headers={"X-Robots-Tag":["none"],"Cache-Control":["no-cache, private"],"Content-Type":["text/html; charset=UTF-8"],"Server":["Caddy","Apache"],"Location":["http://firefly.lan/login"],"X-Content-Type-Options":["nosniff"],"X-Frame-Options":["deny"],"Content-Security-Policy":["default-src 'none'; object-src 'none'; script-src 'unsafe-eval' 'strict-dynamic' 'self' 'unsafe-inline' 'nonce-ttopMhcw6VewhWtuJnKFcQ=='   ; style-src 'unsafe-inline' 'self'; base-uri 'self'; font-src 'self' data:; connect-src 'self' ; img-src data: 'strict-dynamic' 'self' *.tile.openstreetmap.org ; manifest-src 'self'; form-action 'self'"],"Feature-Policy":["geolocation 'none'; midi 'none'; sync-xhr 'self'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; fullscreen 'self'; payment 'none'"],"X-Permitted-Cross-Domain-Policies":["none"],"X-Xss-Protection":["1; mode=block"],"Date":["Tue, 13 Feb 2024 14:13:01 GMT"],"Referrer-Policy":["no-referrer"],"Set-Cookie":[]} status=302

Machine 2

DBG | ts=1707833630.9788237 logger=http.handlers.reverse_proxy msg=selected upstream dial=firefly_app:8080 total_upstreams=1 
DBG | ts=1707833630.9994383 logger=http.handlers.reverse_proxy msg=upstream roundtrip upstream=firefly_app:8080 duration=0.020565598 request={"remote_ip":"192.168.178.2","remote_port":"43592","client_ip":"192.168.178.2","proto":"HTTP/1.1","method":"GET","host":"firefly.lan:80","uri":"/","headers":{"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["192.168.178.2"],"X-Forwarded-Host":["firefly.lan:80"],"X-Forwarded-Proto":["http"],"Accept-Encoding":["gzip"]}} headers={"Location":["http://firefly.lan/login"],"Date":["Tue, 13 Feb 2024 14:13:50 GMT"],"X-Content-Type-Options":["nosniff"],"X-Robots-Tag":["none"],"Feature-Policy":["geolocation 'none'; midi 'none'; sync-xhr 'self'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; fullscreen 'self'; payment 'none'"],"Server":["Apache"],"Cache-Control":["no-cache, private"],"Content-Security-Policy":["default-src 'none'; object-src 'none'; script-src 'unsafe-eval' 'strict-dynamic' 'self' 'unsafe-inline' 'nonce-xNFZuJvEoAAxYb8OypejLw=='   ; style-src 'unsafe-inline' 'self'; base-uri 'self'; font-src 'self' data:; connect-src 'self' ; img-src data: 'strict-dynamic' 'self' *.tile.openstreetmap.org ; manifest-src 'self'; form-action 'self'"],"Referrer-Policy":["no-referrer"],"Set-Cookie":[],"X-Frame-Options":["deny"],"X-Xss-Protection":["1; mode=block"],"X-Permitted-Cross-Domain-Policies":["none"],"Content-Type":["text/html; charset=UTF-8"]} status=302 
DBG | ts=1707833631.0066507 logger=http.handlers.reverse_proxy msg=selected upstream dial=firefly_app:8080 total_upstreams=1 
DBG | ts=1707833631.0295167 logger=http.handlers.reverse_proxy msg=upstream roundtrip upstream=firefly_app:8080 duration=0.02281152 request={"remote_ip":"192.168.178.5","remote_port":"49892","client_ip":"192.168.178.5","proto":"HTTP/1.1","method":"GET","host":"firefly.lan","uri":"/login","headers":{"User-Agent":["curl/7.81.0"],"Accept":["*/*"],"X-Forwarded-For":["192.168.178.5"],"X-Forwarded-Proto":["http"],"X-Forwarded-Host":["firefly.lan"]}} headers={"Content-Type":["text/html; charset=UTF-8"],"Date":["Tue, 13 Feb 2024 14:13:51 GMT"],"Content-Security-Policy":["default-src 'none'; object-src 'none'; script-src 'unsafe-eval' 'strict-dynamic' 'self' 'unsafe-inline' 'nonce-6shmeEsWfKAeTuUoFNUw2w=='   ; style-src 'unsafe-inline' 'self'; base-uri 'self'; font-src 'self' data:; connect-src 'self' ; img-src data: 'strict-dynamic' 'self' *.tile.openstreetmap.org ; manifest-src 'self'; form-action 'self'"],"X-Xss-Protection":["1; mode=block"],"X-Permitted-Cross-Domain-Policies":["none"],"Server":["Apache"],"X-Frame-Options":["deny"],"X-Robots-Tag":["none"],"Cache-Control":["no-cache, private"],"X-Content-Type-Options":["nosniff"],"Referrer-Policy":["no-referrer"],"Feature-Policy":["geolocation 'none'; midi 'none'; sync-xhr 'self'; microphone 'none'; camera 'none'; magnetometer 'none'; gyroscope 'none'; fullscreen 'self'; payment 'none'"],"Set-Cookie":[],"Vary":["Accept-Encoding"]} status=200

3. Caddy version:

2.7.6

4. How I installed and ran Caddy:

a. System environment:

Ubuntu, Docker

c. Service/unit/compose file:

I am using Portainer, here the relevant settings:
Published Ports: 80->80 | 443->443
Command: 'caddy' 'run' '--config' '/etc/caddy/Caddyfile' '--adapter' 'caddyfile'
Volumes:
- /config -> /docker/caddy/config
- /data -> /docker/caddy/data
- /etc/caddy/Caddyfile -> /docker/caddy/Caddyfile
Network: internal (bridged)

d. My complete Caddy config:

Machine 1

#firefly
ff.arianw.de	{
	reverse_proxy http://firefly.lan {
	header_up Host {upstream_hostport}
	}
}

#kimai
ki.arianw.de {
	reverse_proxy http://kimai.lan {
		header_up Host {upstream_hostport}
	}
}

Machine 2

#firefly
http://firefly.lan	{
	reverse_proxy firefly_app:8080
}

#kimai
http://kimai.lan	{
	reverse_proxy kimai-nginx:80
}

Thank you for your time.

2

It doesn’t say “only”, it says “often”:

The reason is because the Host header is passed through as-is from the original request, unless you override it.

On your upstream Caddy, you’re using a host matcher (i.e. http://kimai.lan) so it only handles request that match.

If you instead change your upstream to use http://ki.arianw.de, matching the original request, then there would be no need for header_up Host {upstream_hostport}.

If you still need local access to work, you can keep both domains, so http://kimai.lan, http://ki.arianw.de as your site address.

3

Okay, that illuminated me quite a bit. Thanks.

I still get the problem from firefly that it reports the base-uri as http://ff.arianw.de and not https://ff.arianw.de
This leads to an unusable site because of security requirements.

Is this related to firefly or caddy?

I changed the caddyfiles to

Machine 1

#firefly
ff.arianw.de	{
	reverse_proxy http://firefly.lan {
	}
}

Machine 2

#firefly
http://firefly.lan, http://ff.arianw.de	{
	reverse_proxy firefly_app:8080
}

I also tested setting it to https://ff.arianw.de and ff.arianw.de. Neither worked and resulted in redirection errors.

4

In your 2nd machine, you’ll need to configure trusted_proxies, otherwise headers like X-Forwarded-Proto won’t get passed through correctly. I assume the app is using that header to build its URLs.

1 Like
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.