Cloudflare Proxy & X-Forwarded-For / Client IP Issue

Help
1

1. The problem I’m having:

Unable to get “X-Forwarded-For” to show correct IP on a whoami webpage, also unable to get client_ip to show correctly. It is showing Cloudflare IP despite building Caddy with the Cloudflare Module, and amending the Caddyfile to include what I believe are the correct elements.

2. Error messages and/or full log output:

{"level":"info","ts":1723413883.4145572,"logger":"http.log.access.log7","msg":"handled request","request":{"remote_ip":"172.71.26.33","remote_port":"34842","client_ip":"172.71.26.33","proto":"HTTP/2.0","method":"GET","host":"whoami.allthatlab.xyz","uri":"/","headers":{"Accept-Encoding":["gzip, br"],"Cf-Ray":["8b1b85e30fed71e1-LHR"],"X-Forwarded-Proto":["https"],"Accept":["*/*"],"Cf-Ipcountry":["GB"],"X-Forwarded-For":["82.132.236.243"],"Cf-Visitor":["{\"scheme\":\"https\"}"],"User-Agent":["iCurlHTTP/1.17 libcurl/7.83.1 OpenSSL/1.1.1o zlib/1.2.11 nghttp2/1.47.0"],"Cf-Connecting-Ip":["82.132.236.243"],"Cdn-Loop":["cloudflare"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"whoami.allthatlab.xyz"}},"bytes_read":0,"user_id":"","duration":0.002957776,"size":567,"status":200,"resp_headers":{"Content-Encoding":["gzip"],"Content-Type":["text/plain; charset=utf-8"],"Date":["Sun, 11 Aug 2024 22:04:43 GMT"],"Vary":["Accept-Encoding"],"Server":["Caddy"],"Alt-Svc":["h3=\":443\"; ma=2592000"]}}

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

a. System environment:

Unraid, Docker

b. Build:

FROM caddy:2.8-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/hslatman/caddy-crowdsec-bouncer/crowdsec \
    --with github.com/WeidiDeng/caddy-cloudflare-ip

FROM caddy:2.8

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

c. Service/unit/compose file:

docker run
  -d
  --name='caddy'
  --net='farrosphere'
  --pids-limit 2048
  -e TZ="Europe/London"
  -e HOST_OS="Unraid"
  -e HOST_HOSTNAME="farrosphere"
  -e HOST_CONTAINERNAME="caddy"
  -e 'CERT_EMAIL'='admin@allthatlab.xyz'
  -e 'CF_API_TOKEN'=''
  -e 'CROWDSEC_API'=''
  -l net.unraid.docker.managed=dockerman
  -l net.unraid.docker.icon='https://d1q6f0aelx0por.cloudfront.net/product-logos/library-caddy-logo.png'
  -p '2080:80/tcp'
  -p '2443:443/tcp'
  -p '2443:443/udp'
  -v '/mnt/user/appdata/caddy/data':'/data':'rw'
  -v '/mnt/user/appdata/caddy/config':'/config':'rw'
  -v '/mnt/user/appdata/caddy/config/Caddyfile':'/etc/caddy/Caddyfile':'rw'
  -v '/mnt/user/dmz/crowdsec':'/var/log/crowdsec':'rw' 'caddy-allthatlab'

d. My complete Caddy config:

{
	acme_dns cloudflare {env.CF_API_TOKEN}
	email {env.CERT_EMAIL}

	debug

	crowdsec {
		api_key {env.CROWDSEC_API}
		api_url http://crowdsec:8080
	}

	servers {
		trusted_proxies cloudflare
		trusted_proxies static private_ranges 100.64.0.0/10
		client_ip_headers Cf-Connecting-Ip X-Forwarded-For
	}

	admin :2019
}

(essentials) {
	encode gzip
	
	tls {
		dns cloudflare {env.CF_API_TOKEN}
		resolvers 1.1.1.1
	}

	log {
		output file /var/log/crowdsec/caddy.log
	}
}

(security_headers) {
	header_up Strict-Transport-Security "max-age=31536000;"
	header_up X-XSS-Protection "1; mode=block"
	header_up X-Frame-Options "SAMEORIGIN"
	header_up X-Robots-Tag "noindex, nofollow"
	header_up X-Content-Type-Options "nosniff"
	header_up -Server
	header_up -X-Powered-By
	header_up Referrer-Policy "same-origin"
}

(authelia) {
	forward_auth authelia:9091 {
		uri /api/authz/forward-auth
		copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
		import security_headers
	}
}

import /config/*.caddy

---

# Whoami (Auth)
whoami.allthatlab.xyz {
	import essentials
	import authelia

	log {
		output file /var/log/crowdsec/whoami.log
	}

	reverse_proxy whoami:80 {
		import security_headers
	}
}

5. Links to relevant resources:

Curl Response:

Name: All That Lab
Hostname: 565671e3cbf7
IP: 127.0.0.1
IP: ::1
IP: 172.19.0.60
RemoteAddr: 172.19.0.77:36238
GET / HTTP/1.1
Host: whoami.allthatlab.xyz
User-Agent: iCurlHTTP/1.17 libcurl/7.83.1 OpenSSL/1.1.1o zlib/1.2.11 nghttp2/1.47.0
Accept: */*
Accept-Encoding: gzip, br
Cdn-Loop: cloudflare
Cf-Connecting-Ip: 82.132.236.243
Cf-Ipcountry: GB
Cf-Ray: 8b1b85e30fed71e1-LHR
Cf-Visitor: {"scheme":"https"}
Referrer-Policy: same-origin
Remote-Email: {http.reverse_proxy.header.Remote-Email}
Remote-Groups: {http.reverse_proxy.header.Remote-Groups}
Remote-Name: {http.reverse_proxy.header.Remote-Name}
Remote-User: {http.reverse_proxy.header.Remote-User}
Strict-Transport-Security: max-age=31536000;
X-Content-Type-Options: nosniff
X-Forwarded-For: 172.71.26.33
X-Forwarded-Host: whoami.allthatlab.xyz
X-Forwarded-Proto: https
X-Frame-Options: SAMEORIGIN
X-Robots-Tag: noindex, nofollow
X-Xss-Protection: 1; mode=block
2

Howdy @Heavensong89, welcome to the Caddy community.

I think this is your issue. You can’t specify multiple trusted_proxies in this manner - I believe only the latest would apply. That means you’ve overridden your cloudflare trust with private ranges, making Cloudflare untrusted again.

A quick search found this post: Multiple trusted_proxies directives - #2 by francislavoie

Which links to this module: GitHub - fvbommel/caddy-combine-ip-ranges: IP prefix module for Caddy that combines the output of other IP prefix modules.

You should be able to use that to combine the static and cloudflare IP source modules together.

3 Likes
3

Amazing thanks! Updated the servers section in my global block, all working and can now set an X-Real-Ip header based on the client IP, works internally, via Tailscale and also accessing via Internet.

servers {
		trusted_proxies combine {
			cloudflare
			static private_ranges 100.64.0.0/10
		}
		client_ip_headers Cf-Connecting-Ip X-Forwarded-For
	}

...

(security_headers) {
	header_up Strict-Transport-Security "max-age=31536000;"
	header_up X-XSS-Protection "1; mode=block"
	header_up X-Frame-Options "SAMEORIGIN"
	header_up X-Robots-Tag "noindex, nofollow"
	header_up X-Content-Type-Options "nosniff"
	header_up -Server
	header_up -X-Powered-By
	header_up Referrer-Policy "same-origin"
	header_up X-Real-Ip {client_ip}
}
2 Likes
4

This stuff doesn’t make sense btw. Most of those are response headers, but you’re adding it to the request being sent upstream.

Are you sure your upstream needs X-Real-Ip? Caddy passes through X-Forwarded-For with the client IP already. Your upstream should probably just use that header instead.

This setup could still be insecure btw. You would need to make sure that your downstream proxy at 100.64.0.0/10 guarantees that it removes any Cf-Connecting-Ip header, otherwise someone making a request through that server could add Cf-Connecting-Ip to trick your server in thinking it used any IP address.

For the Cloudflare case it’s safe because Cloudflare guarantees Cf-Connecting-Ip will be set, even though they don’t properly protect X-Forwarded-For from spoofing, but since you have Cf-Connecting-Ip ordered first in your config then it will always ignore X-Forwarded-For (good in this case).

2 Likes
5

Re the Headers, if they’re not needed they can be removed, they were taken from a setup guide I was reading through, however that was using header rather than header_up or header_down, so perhaps I’ve configured that part incorrectly! X-Real-Ip could certainly be removed yes I don’t actually think anything uses it, I was just used to seeing it from my days using Traefik.

On the Trusted proxies - the 100.64.0.0/10 range is for Tailscale that uses it’s own DNS rewriting the traffic locally, so it doesn’t hit Cloudlfare Proxied records at any point.

6

The vulnerability isn’t someone going through Tailscale and then through Cloudflare.

Lets imagine you have a reverse proxy elsewhere in your Tailnet. That reverse proxy is accessible from the internet. That reverse proxy is transparent and does not moderate which headers it forwards along to Caddy. This request path does not involve Cloudflare at all.

An attacker could craft a request e.g. curl --header "Cf-Connecting-Ip: <some-trusted-IP>" https://tailnetproxy.example.com. Your reverse proxy on the Tailnet would forward it to Caddy, including the crafted header. Caddy would trust the connecting proxy because you’ve configured it to trust the whole Tailnet, and it will therefore trust the attacker’s Cf-Connecting-Ip. The attacker could thereby fool your Caddy server into thinking the original client was any IP address, including IP addresses that may bypass security mechanisms or position the attacker advantageously, e.g. appearing as though within your LAN if that would give them access to things they shouldn’t have.

This is only a problem if you do in fact have a reverse proxy in front of Caddy somewhere in the Tailnet, and it is also accessible to threat actors (such as being available to the internet), and it does not scrub Cf-Connecting-Ip from requests, and you have some website on Caddy which would be more vulnerable to attacks from specific IP addresses, or perhaps some mechanism like fail2ban which could be abused to deny service to legitimate IPs.

It’s a limited attack path, but it exists and it’s worth noting, because when it does strike, it can really suck.

1 Like
7

Hmm okay, so I’ve removed client_ip_headers because XFF is the default entry anyway, and that is being correctly written now.

I do have my Tailnet setup to only use a small IP range, I could amend my trusted proxies above to that smaller range. I also don’t believe there is any other reverse proxy, I use Caddy both internally and externally, and only have DNS records published via Cloudflare for the services I want accessed on the web anyway. Internally Adguard rewrites the FQDN to the server IP, on the web it’s cloudflare dns.

8

I will note that internal/external DNS doesn’t really matter to an attacker as they can craft a request to any IP address for any host; the only question is network access. DNS is a tool of convenience, not a necessary step.

Do you really need to configure trusted proxies from 100.64.0.0/10 at all then, if you don’t have any reverse proxies there?

This configuration is not for clients. It’s for additional fronting reverse proxies (like Cloudflare is). Regular clients don’t send X-Forwarded-For, they just come straight from the relevant IP address and connect straight to Caddy.

1 Like
9

That’s a good point, I’ve perhaps just wrongly assumed that they needed to be trusted. I’ve just removed that range and reloaded, doesn’t look to have had any impact.

Network security is mind boggling!

1 Like
10

It can get very complicated very quickly!

Generally speaking, though, most of these kinds of things are introduced to solve a specific problem. For direct client access (stuff in your Tailnet talking staight to Caddy) there’s no problem to solve. It’s only when you have clients connecting to Cloudflare and Cloudflare connecting to Caddy that you have a new problem to solve - how to (in a secure manner) log where the requests actually came from! Because any client could simply lie about it in headers. Only Cloudflare knows the truth, so you have to decide to trust Cloudflare.

It’s a great big ball of problems and solutions on problems and solutions on problems and solutions and it can get hard to track sometimes. That’s one of the reasons we put such a high value on simplicity - of configuration, of complexity, of reproducibility.

1 Like
11

I wish I was the kind of person that could just decide okay this isn’t for me then, but alas I’ve fallen down the self-hosted rabbit hole now.

Revised snippet below, I think this covers what’s been mentioned so far plus I added trusted_proxies_strict as I saw that referenced in the Caddy docs if you’re using something like AWS or Cloudflare.

I did amend the header_up to header_down in my headers section, but will completely revisit the that to see what/if any are needed. I kept X-Real-Ip just whilst I was looking at whoami but will likely remove it!

{
	acme_dns cloudflare {env.CF_API_TOKEN}
	email {env.CERT_EMAIL}

	crowdsec {
		api_key {env.CROWDSEC_API}
		api_url http://crowdsec:8080
	}

	servers {
		trusted_proxies combine {
			cloudflare
			static private_ranges
		}
		trusted_proxies_strict
	}
	admin :2019

	order filter after encode
}

...


(security_headers) {
	header_down Strict-Transport-Security "max-age=31536000;"
	header_down X-XSS-Protection "1; mode=block"
	header_down X-Frame-Options "SAMEORIGIN"
	header_down X-Robots-Tag "noindex, nofollow"
	header_down X-Content-Type-Options "nosniff"
	header_down -Server
	header_down -X-Powered-By
	header_up -Accept-Encoding # Used by Theme Park
	header_down Referrer-Policy "same-origin"
	header_up X-Real-Ip {client_ip}
}
1 Like
12

Just to chime in with similar reasoning to the tailnet subnet trust, you might want to also consider the relevance of static private_ranges too?

While it may be unlikely, any client that connects to Caddy within those trusted subnets are also going to have any such headers trusted. If you can be more explicit about the IP to trust when required, that’s usually preferrable.

Notably with subnet wide trust, you can sometimes run into connections that are routed through the subnet gateway IP (eg: 172.16.0.1), which is a common issue to see with Docker (at least prior to v27 IIRC, any IPv6 connection to the host by default would route to IPv4-only container IP with a client IP as the gateway IP). That allowed such untrustworthy clients to become trusted even though they were not actually part of the subnet.

1 Like
13

Actually, you should probably keep it, because like I said earlier, Cloudflare does not protect X-Forwarded-For from spoofing by default. See Forwarded Headers | Integration | Authelia which explains how you can fix that in your Cloudflare settings, or you can use client_ip_headers to make Cloudflare’s header get taken first… as long as there’s no other path to your server from sources you don’t trust.

1 Like
14

I had a similar thought as the discussion around the tailnet range was happening. In previous reverse proxies, I only trusted the 192.168 private subnet as opposed to all private ranges, I think I just saw that private_ranges was available and went with that instead.

15

Thanks for this - I have followed those steps in Authelia already, but I suppose there is no harm in keeping the client_ip_headers Cf-Connecting-Ip X-Forwarded-For in place.

16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.