Confused about the use of trusted proxies

1. The problem I’m having:

I have a VPS with IP x.y.z.w. This VPS has a Wireguard tunnel to my home. The tunnel uses the block 10.1.1.0/24. I am running two caddy instances, 1 on the VPS and another at my home.

I access a site hosted on Caddy instance on VPS. This instance adds X-Forwarded-For, X-Real-IP to requests before forwarding them to the caddy instance hosted at my home.

The instance at home forwards the request to the right service. The instance at home has these values in the global trusted_proxies field.

 "trusted_proxies": {
     "ranges": [
        "10.1.1.0/24",
     ],
     "source": "static"
 }

X-Forwarded-Forheader is overridden to the internal wireguard IP of VPS. If my understanding is correct, It should be trusting the X-Forwarded-For header set by the Caddy instance on the VPS and not write over those values ?

2. Error messages and/or full log output:

Logs from the Caddy instance at home showing the Caddy instance on VPS set the headers correctly.

{"level":"info","ts":1735244287.951898,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"10.1.1.3","remote_port":"36812","client_ip":"27.7.96.110","proto":"HTTP/2.0","method":"POST","host":"jellyfin.ishanjain.me","uri":"/Sessions/Playing/Progress","headers":{"User-Agent":["Ktor client"],"Content-Type":["application/json"],"X-Real-Ip":["27.7.96.110"],"Authorization":[],"Accept-Encoding":["gzip"],"Accept-Charset":["UTF-8"],"Accept":["application/json, application/octet-stream;q=0.9, */*;q=0.8"],"Content-Length":["290"],"X-Forwarded-Host":["jellyfin.ishanjain.me"],"X-Forwarded-For":["27.7.96.110"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"jellyfin.ishanjain.me"}},"bytes_read":290,"user_id":"","duration":0.002382812,"size":0,"status":204,"resp_headers":{"X-State":["KA"],"X-City":["Bengaluru"],"X-Country":["IN"],"Alt-Svc":["h3=\":443\"; ma=2592000"],"X-Response-Time-Ms":["1.9906"],"Date":["Thu, 26 Dec 2024 20:18:07 GMT"]}}

The http request as it was received by the service shows both X-Forwarded-Forand X-Real-IP were set to the Wireguard IP of the VPS.

(Captured using wireshark on the host running the service)

3. Caddy version:

2.7.6

4. How I installed and ran Caddy:

Built using xcaddy

a. System environment:

Hosted on Debian 11 using systemd

b. Command:

c. Service/unit/compose file:

d. My complete Caddy config:

5. Links to relevant resources:

Use the latest version, v2.8.4.

Show your full config.

Upgraded to the latest version of Caddy.

Here are my configs. They are a bit long so I used pastebin.

Caddy config for the instance on VPS https://paste.debian.net/plainh/f6c7658b
This sets the X-Forwarded-For headers with the correct client address before sending them to the caddy instance at home.

Caddy config for the instance at home https://paste.debian.net/plainh/b0a8d709
This instance has the IP address 10.0.50.3 and this instance is overwriting the x-forwarded-for header set by the other instance despite the IP address of other instance being present in the trusted_proxies config

Hi, Bumping up this thread. I am still looking for a solution to this problem.

Do I have to import the trusted_proxies config in the interior blocks somehow to make this work?

Documentation says client_ip is derived from an upstream’s X-Forwarded-For header if that upstream’s address is in the trusted_proxies list.

In the logs of caddy instance at home, I see the correct client_ip. Why on earth is it then rewriting it to the IP address of upstream proxy(hosted on the VPS) before sending the request to the service ??

this is so confusing

Edit #1:
Removing all trusted_proxies sections(local and global) from the file yields the expected result. client_ip is set to upstream proxy ip and the x-forwarded-for header set by upstream proxy is completely discarded.

Either I am doing some thing incredibly stupid or caddy is annoyingly buggy.

I thought maybe my custom modules were causing problems so I tried a stock build of the latest version and the exact same behavior! why??

It also doesn’t help to add 0.0.0.0/0 and ::/0 to the trusted_proxies config. Why else is it doing this ??

Custom modules list

caddy.listeners.layer4
dns.providers.cloudflare
exec
geoip2
http.authentication.providers.authorizer
http.handlers.authenticator
http.handlers.exec
http.handlers.geoip2
http.handlers.waf
layer4
layer4.handlers.echo
layer4.handlers.proxy
layer4.handlers.proxy_protocol
layer4.handlers.socks5
layer4.handlers.subroute
layer4.handlers.tee
layer4.handlers.throttle
layer4.handlers.tls
layer4.matchers.clock
layer4.matchers.dns
layer4.matchers.http
layer4.matchers.local_ip
layer4.matchers.not
layer4.matchers.openvpn
layer4.matchers.postgres
layer4.matchers.proxy_protocol
layer4.matchers.quic
layer4.matchers.rdp
layer4.matchers.regexp
layer4.matchers.remote_ip
layer4.matchers.socks4
layer4.matchers.socks5
layer4.matchers.ssh
layer4.matchers.tls
layer4.matchers.winbox
layer4.matchers.wireguard
layer4.matchers.xmpp
layer4.proxy.selection_policies.first
layer4.proxy.selection_policies.ip_hash
layer4.proxy.selection_policies.least_conn
layer4.proxy.selection_policies.random
layer4.proxy.selection_policies.random_choose
layer4.proxy.selection_policies.round_robin
security
tls.handshake_match.alpn

  Non-standard modules: 45

Your own config of the home instance has this:

{
  "handler": "headers",
  "request": {
    "set": {
      "X-Forwarded-For": [
        "{http.request.remote.host}"
      ],
      "X-Real-IP": [
        "{http.request.remote.host}"
      ]
    }
  }
  ...

Why are you manipulating X-Forwarded-For? If you configure it manually, Caddy will do what you told it, otherwise, it’ll just do the right thing on its own.

Good eye! Thank you!!

is there a way to conditionally set this header? i.e. It should set it if the headers are not present in the request and not do any thing if they are already present ?

Use matchers. Matchers are conditions for actions. For absence of header field, consider this example:

"match": [
	{
		"header": {
				"X-Forwarded-For": null      
		}
	}
]

Per the docs:

If a list is null, the header must not exist.
JSON Config Structure - Caddy Documentation

By the way, why are you using the JSON format? It’s more user friendly to use Caddyfile. The JSON format is more apt for automation-driven config changes.

Use matchers. Matchers are conditions for actions. For absence of header field, consider this example:

Thank you

By the way, why are you using the JSON format? It’s more user friendly to use Caddyfile. The JSON format is more apt for automation-driven config changes.

My config is not super simple and writing it in Caddyfile format is quite confusing plus when I started using caddy, There was very little documentation on Caddyfile format beyond the basics. The JSON format is much easier to reason about so I use that

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.