X-Forwarded-For header from Haproxy is (discarded?) by Caddy. Same setup works with Nginx

[tokokev139]

1. Caddy version (caddy version):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

haproxy on 80/443 which is handed off to caddy 8080/8443

a. System environment:

Ubuntu 20.04

b. Command:

Systemd?

systemctl start/stop/etc caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=exec
Group=exec
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
        http_port 8080
        https_port 8443
}

x.y.z {
        root * /server/web-server/public-html/x.y.z/
        php_fastcgi unix//run/php/php7.4-fpm.sock
        file_server
}

3. The problem I’m having:

Hi there.

I have a setup where haproxy (80/443) is in-front of caddy (127.0.0.1:8080/8443)

I am trying to make it so that caddy knows the ‘Real IP’ of the request, passed through haproxy, using the X-Forwarded-For header.

It seems that Caddy does not ‘pass through’ this header AND/OR discards this header AND/OR does something where, when I attempt to print $_SERVER['X-Forwarded-For'] with php-fpm, I get the wrong answer.

HTTP_X_FORWARDED_FOR=127.0.0.1

If I replace / hot-swap Caddy with Nginx, and print the same header, using the same copy of php-fpm wired up to Nginx, then the header is passed through correctly. (i.e. the ‘Real IP’ of that request, not the 127.0.0.1 ‘proxy_passed’ localhost address)

HTTP_X_FORWARDED_FOR=185.223.XXX.X

Here is my crude haproxy configuration

frontend frontend-http
	bind 						*:80
	mode						http
	default_backend 			backend-caddy-http
	#option 					httpclose
	option 						forwardfor
	# reqidel 					^X-Forwarded-For:.*
	# http-request add-header 	X-Forwarded-For %[src]

frontend frontend-https
	bind						*:443
	mode						tcp
	default_backend				backend-caddy-https
	#option 					httpclose
	option 						forwardfor
	# reqidel 					^X-Forwarded-For:.*
	# http-request add-header 	X-Forwarded-For %[src]

backend backend-caddy-http
	mode						http
	server						frontend-http 127.0.0.1:8080  # Caddy/Nginx HTTP on 8080
	#option 					httpclose
	option 						forwardfor
	# reqidel 					^X-Forwarded-For:.*
	# http-request add-header 	X-Forwarded-For %[src]

backend backend-caddy-https
	mode						tcp
	server						frontend-https 127.0.0.1:8443  # Caddy/Nginx HTTPS on 8443
	#option 					httpclose
	option 						forwardfor
	# reqidel 					^X-Forwarded-For:.*
	# http-request add-header 	X-Forwarded-For %[src]

Is this a bug? Any pointers, advise or links to documentation explaining what Caddy v2 is doing with with the request headers in-between the time Caddy receives the request from Haproxy, and the time it takes to print the header out to screen, would be greatly appreciated.

Also, setting the log level to DEBUG and looking at the Caddy logs, when doing a request, it doesn’t show the X-Forwarded headers in the request.headers section.

{
    "level": "info",
    "ts": 1629922024.871911,
    "logger": "http.log.access.log0",
    "msg": "handled request",
    "request": {
        "remote_addr": "127.0.0.1:60754",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "x.y.z",
        "uri": "/",
        "headers": {
            "Cache-Control": ["max-age=0"],
            "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],
            "Sec-Fetch-Site": ["none"],
            "Sec-Fetch-Mode": ["navigate"],
            "Sec-Fetch-User": ["?1"],
            "Sec-Fetch-Dest": ["document"],
            "Sec-Ch-Ua": ["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"92\""],
            "Sec-Ch-Ua-Mobile": ["?0"],
            "Dnt": ["1"],
            "Upgrade-Insecure-Requests": ["1"],
            "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.134 Safari/537.36"],
            "Accept-Encoding": ["gzip, deflate, br"],
            "Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"]
        },
        "tls": {
            "resumed": true,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "x.y.z"
        }
    },
    "common_log": "127.0.0.1 - - [25/Aug/2021:20:07:04 +0000] \"GET / HTTP/2.0\" 200 1876",
    "duration": 0.000758017,
    "size": 1876,
    "status": 200,
    "resp_headers": {
        "Server": ["Caddy"],
        "Content-Type": ["text/html; charset=UTF-8"]
    }
}{
    "level": "info",
    "ts": 1629922025.0205097,
    "logger": "http.log.access.log0",
    "msg": "handled request",
    "request": {
        "remote_addr": "127.0.0.1:60754",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "x.y.z",
        "uri": "/favicon.ico",
        "headers": {
            "Pragma": ["no-cache"],
            "Cache-Control": ["no-cache"],
            "Sec-Ch-Ua": ["\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"92\""],
            "Dnt": ["1"],
            "Sec-Ch-Ua-Mobile": ["?0"],
            "Accept": ["image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8"],
            "Sec-Fetch-Mode": ["no-cors"],
            "Sec-Fetch-Dest": ["image"],
            "Referer": ["https://x.y.z"],
            "Accept-Language": ["en-GB,en-US;q=0.9,en;q=0.8"],
            "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.134 Safari/537.36"],
            "Sec-Fetch-Site": ["same-origin"],
            "Accept-Encoding": ["gzip, deflate, br"]
        },
        "tls": {
            "resumed": true,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "proto_mutual": true,
            "server_name": "x.y.z"
        }
    },
    "common_log": "127.0.0.1 - - [25/Aug/2021:20:07:05 +0000] \"GET /favicon.ico HTTP/2.0\" 200 1818",
    "duration": 0.000800669,
    "size": 1818,
    "status": 200,
    "resp_headers": {
        "Server": ["Caddy"],
        "Content-Type": ["text/html; charset=UTF-8"]
    }
}

Thanks.

[francislavoie]

Are you sure your HAproxy setup is sending the header? If it was sending it, it would should in the request headers in the logs.

I don’t think Caddy is discarding it, I think Caddy is just not receiving it.

For more details of what’s happening, you can turn on the debug global option, then check your logs with this command:

journalctl -u caddy --no-pager | less +G

[tokokev139]

Hi Francis,

I had this in my Caddyfile before (didn’t think much of it,) which may have hidden some headers before, in my original post. (i.e. may have overridden the ‘debug’ directive?)

log {
  output file     /var/log/caddy/everything
  format          console
  level           info
}

Using the journalctl command you posted, it does show an X-Forwarded-For header of 127.0.0.1.

Now I see the X-Forwarded header is of 127.0.0.1 which is incorrect.

I would say this is a misconfiguration of my haproxy configuration, but I don’t understand why it works with Nginx of which it can resolve the correct IP (as all I’m doing is the same thing / proxy_passing incoming traffic from *:80, *:443 to 127.0.0.1:8080, 8443 therefore Nginx, like Caddy, is only seeing an incoming request from 127.0.0.1 and therefore should be functionally analogous)

Confused.

Anyway, going to sleep on this, thanks for the reply.

[francislavoie]

X-Forwarded-For: 127.0.0.1 will be what Caddy is setting itself when proxying to your fastcgi upstream, as documented here:

reverse_proxy (Caddyfile directive) - Caddy Documentation

To prove that I’m sure this is an issue with Haproxy simply not sending the header, I can try this config:

{
        debug
}

:8881 {
	log
	reverse_proxy https://www.google.com {
	        header_up Host {http.reverse_proxy.upstream.hostport}
	}
}

Running this, Caddy will output access logs and debug logs to stderr (console output).

Then, we can make a request with curl, adding the X-Forwarded-For header to the request, to roughly mimic what Haproxy should be doing:

$ curl -v -H "X-Forwarded-For: 123.123.123.123" localhost:8881
* Rebuilt URL to: localhost:8881/
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 8881 (#0)
> GET / HTTP/1.1
> Host: localhost:8881
> User-Agent: curl/7.55.1
> Accept: */*
> X-Forwarded-For: 123.123.123.123
>
< HTTP/1.1 200 OK
< Cache-Control: private, max-age=0
< Content-Type: text/html; charset=ISO-8859-1
< Date: Thu, 26 Aug 2021 00:58:39 GMT
< Expires: -1
< P3p: CP="This is not a P3P policy! See g.co/p3phelp for more info."
< Server: Caddy
< Server: gws
< Set-Cookie: <snip for brevity>
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 0
< Transfer-Encoding: chunked
<
// rest omitted, just the HTML response from Google

So you can see here that Caddy successfully proxied a request to Google’s homepage, and we do see in the request that there’s an X-Forwarded-For: 123.123.123.123 there being sent to Caddy.

Now we look at Caddy’s logs:

2021/08/26 01:00:17.260 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "www.google.com:443", "request": {"remote_addr": "[::1]:57592", "proto": "HTTP/1.1", "method": "GET", "host": "www.google.com:443", "uri": "/", "headers": {"X-Forwarded-Proto": ["http"], "User-Agent": ["curl/7.55.1"], "Accept": ["*/*"], "X-Forwarded-For": ["123.123.123.123, ::1"]}}, "headers": {"Alt-Svc": ["h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000,h3-T051=\":443\"; ma=2592000,h3-Q050=\":443\"; ma=2592000,h3-Q046=\":443\"; ma=2592000,h3-Q043=\":443\"; ma=2592000,quic=\":443\"; ma=2592000; v=\"46,43\""], "Date": ["Thu, 26 Aug 2021 01:00:17 GMT"], "Expires": ["-1"], "Content-Type": ["text/html; charset=ISO-8859-1"], "P3p": ["CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\""], "X-Frame-Options": ["SAMEORIGIN"], "Set-Cookie": [<snip for brevity>], "Cache-Control": ["private, max-age=0"], "Server": ["gws"], "X-Xss-Protection": ["0"]}, "status": 200}
2021/08/26 01:00:17.261 INFO    http.log.access handled request {"request": {"remote_addr": "[::1]:57592", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8881", "uri": "/", "headers": {"User-Agent": ["curl/7.55.1"], "Accept": ["*/*"], "X-Forwarded-For": ["123.123.123.123"]}}, "common_log": "::1 - - [25/Aug/2021:21:00:17 -0400] \"GET / HTTP/1.1\" 200 14815", "user_id": "", "duration": 0.1846372, "size": 14815, "status": 200, "resp_headers": {"Content-Type": ["text/html; charset=ISO-8859-1"], "P3p": ["CP=\"This is not a P3P policy! See g.co/p3phelp for more info.\""], "Set-Cookie": [<snip for brevity>], "Server": ["Caddy", "gws"], "Date": ["Thu, 26 Aug 2021 01:00:17 GMT"], "Expires": ["-1"], "X-Frame-Options": ["SAMEORIGIN"], "Cache-Control": ["private, max-age=0"], "X-Xss-Protection": ["0"]}}

You can clearly see in the second log message that we have "X-Forwarded-For": ["123.123.123.123"] in the request headers, and we can see that Caddy appended its own value to the X-Forwarded-For header in the first log message with "X-Forwarded-For": ["123.123.123.123, ::1"]

(Note ::1 is IPv6 “localhost”, since curl and caddy are running on the same machine, so from the perspective of Caddy, the request came from “home”).

Properly implemented backend services should first verify that all the addresses after the first in X-Forwarded-For are from trusted sources, and only then, take the first address as the “real client IP”.

From this test, we can pretty clearly see when we compare with your log output that Haproxy is not sending the header. So this isn’t an issue with Caddy, but with your Haproxy config.

I don’t have much experience with Haproxy so I can’t really help there, I don’t know what’s wrong with the config you posted.