File_server httpInclude with X-Forwarded-For?

Help
1

1. The problem I’m having:

Hey :wave:

I would like to use the templates directive to “fill” out data in my HTML files that are served via the file_server.
The data that should be used to fill out the template should come from a different end-point that is accessible via the reverse proxy and that utilizes header data, specifically X-Forwarded-For.
The client IP can be trusted in my environment, and it is used in my application.
From what I understand, it should be possible to use the httpInclude template to run a request to the reverse proxy and insert the data into it.
This seems to work for “plain requests,” but I am having trouble with the headers.
I understand that the X-Forwarded-For header is set when the request passes through the reverse proxy, which I can verify since the correct “data” is returned if I “directly” query the path that is handled by the reverse proxy.
But when the request passes through the file_server first and the reverse proxy is queried indirectly from the file_server the X-Forwarded-For header is not set, or rather dropped.
I assume I am trying to solve the issues with the wrong approach and would appreciate any help :slight_smile:

Maybe a bit more motivation: An alternative would be to have two end-points, where one contains the template rendering engine. This rendering end-point could sit behind the reverse-proxy, parse the X-Forwarded-For header, and copy it with a different name to the HTTP request for the ‘backend’ end-point. But as my template requirements are very simple, I was thinking of trying to solve it directly with Caddy to skip over an additional round-trip to the rendering service.

If this sounds like a bad idea or and abuse of httpInclude, please let me know!

2. Error messages and/or full log output:

2023/09/12 22:08:57.660 INFO    using adjacent Caddyfile
2023/09/12 22:08:57.663 INFO    admin   admin endpoint started  {"address": "localhost:8999", "enforce_origin": false, "origins": ["//localhost:8999", "//[::1]:8999", "//127.0.0.1:8999"]}
2023/09/12 22:08:57.663 INFO    tls.cache.maintenance   started background certificate maintenance {"cache": "0xc0002e4580"}
2023/09/12 22:08:57.663 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{}]}}, "http": {"servers":{"srv0":{"listen":[":8088"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"web"},{"handler":"templates"}]},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rewrite","strip_path_prefix":"/api"}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:3000"}]}]}]}],"match":[{"path":["/api/*"]}]},{"handle":[{"handler":"file_server","hide":["./Caddyfile"]}]}]}],"terminal":true}],"automatic_https":{"skip":["localhost"]},"logs":{"logger_names":{"localhost:8088":""}}}}}}
2023/09/12 22:08:57.663 DEBUG   http    starting server loop    {"address": "[::]:8088", "tls": false, "http3": false}
2023/09/12 22:08:57.663 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/09/12 22:08:57.663 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/kai/.local/share/caddy"}
2023/09/12 22:08:57.663 INFO    tls     finished cleaning storage units
2023/09/12 22:08:57.663 INFO    autosaved config (load with --resume flag){"file": "/home/kai/.config/caddy/autosave.json"}
2023/09/12 22:08:57.663 INFO    serving initial configuration
2023/09/12 22:09:03.362 DEBUG   http.handlers.file_server       sanitized path join        {"site_root": "web", "request_path": "/", "result": "web"}
2023/09/12 22:09:03.363 DEBUG   http.handlers.file_server       located index file {"filename": "web/index.html"}
2023/09/12 22:09:03.363 DEBUG   http.handlers.file_server       opening file       {"filename": "web/index.html"}
2023/09/12 22:09:03.372 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"]}}, "method": "GET", "uri": "/rpc/client"}
2023/09/12 22:09:03.372 DEBUG   http.handlers.reverse_proxy     selected upstream  {"dial": "localhost:3000", "total_upstreams": 1}
2023/09/12 22:09:03.378 DEBUG   http.handlers.reverse_proxy     upstream roundtrip {"upstream": "localhost:3000", "duration": 0.006815538, "request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "/rpc/client", "headers": {"User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"]}}, "headers": {"Content-Type": ["application/json; charset=utf-8"], "Date": ["Tue, 12 Sep 2023 22:09:03 GMT"], "Server": ["postgrest/11.2.0"], "Content-Range": ["0-0/*"]}, "status": 200}
2023/09/12 22:09:03.379 INFO    http.log.access handled request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"]}}, "bytes_read": 0, "user_id": "", "duration": 0.007106501, "size": 163, "status": 200, "resp_headers": {"Server": ["Caddy", "postgrest/11.2.0"], "Date": ["Tue, 12 Sep 2023 22:09:03 GMT"], "Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8"]}}
2023/09/12 22:09:03.379 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"Accept": ["*/*"], "Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"], "User-Agent": ["curl/8.2.1"]}}, "method": "GET", "uri": "/rpc/sample"}
2023/09/12 22:09:03.379 DEBUG   http.handlers.reverse_proxy     selected upstream  {"dial": "localhost:3000", "total_upstreams": 1}
2023/09/12 22:09:03.380 DEBUG   http.handlers.reverse_proxy     upstream roundtrip {"upstream": "localhost:3000", "duration": 0.001410221, "request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "/rpc/sample", "headers": {"Caddy-Templates-Include": ["1"], "User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Accept-Encoding": ["identity"]}}, "headers": {"Date": ["Tue, 12 Sep 2023 22:09:03 GMT"], "Server": ["postgrest/11.2.0"], "Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8", "text/html; charset=utf-8"]}, "status": 200}
2023/09/12 22:09:03.380 INFO    http.log.access handled request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"], "User-Agent": ["curl/8.2.1"], "Accept": ["*/*"]}}, "bytes_read": 0, "user_id": "", "duration": 0.001540195, "size": 280, "status": 200, "resp_headers": {"Server": ["Caddy", "postgrest/11.2.0"], "Date": ["Tue, 12 Sep 2023 22:09:03 GMT"], "Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8", "text/html; charset=utf-8"]}}
2023/09/12 22:09:03.381 INFO    http.log.access handled request {"request": {"remote_ip": "::1", "remote_port": "44822", "client_ip": "::1", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "/", "headers": {"User-Agent": ["curl/8.2.1"], "Accept": ["*/*"]}}, "bytes_read": 0, "user_id": "", "duration": 0.018479596, "size": 1288, "status": 200, "resp_headers": {"Server": ["Caddy"], "Content-Type": ["text/html; charset=utf-8"], "Content-Length": ["1288"]}}

3. Caddy version:

2.7.4

4. How I installed and ran Caddy:

a. System environment:

NixOS (unstable branch)

b. Command:

caddy run

c. Service/unit/compose file:

d. My complete Caddy config:

{
	debug
}
http://localhost:8088 {
	log
	root * web
	handle_path /api/* {
		reverse_proxy localhost:3000
	}
	templates
	file_server
}

5. Links to relevant resources:

2

That’s tricky. httpInclude was designed to just make a simple HTTP request, it wasn’t designed to be similar to a reverse proxy.

@matt WDYT?

3

Hey,

I don’t know if I communicated my intentions clearly.
I do not necessarily need httpInclude to act as a reverse proxy.
What I think I want is a way to set/keep(?) the X-Forwarded-For header when doing a httpInclude request to an API that sits behind a reverse proxy.
Maybe there is a way to set/copy the header before it passes through the httpInclude module (and to keep the header)?

But thank you already for your quick reply! :slight_smile:

4

X-Forwarded-For is inherently a reverse proxy thing. So is passing through headers transparently.

httpInclude currently doesn’t support passing headers, it only takes a URL as input. Maybe it could be changed to take a list of headers as input as well. But I’ll let @matt decide on that.

5

Ah, okay.

Yeah sorry, still quite unexperienced with reverse proxies/caddy :smiley:
I am happy to further clarify any points if necessary :slight_smile:

6

Oh I see, so you want the headers from the original request to be kept with the request used by httpInclude? i.e. you want to pass along the headers too? (Or just one header)

7

Yes, I think this would solve my problem.
If I understand it correctly than by passing the request through a reverse proxy first, then through to the file server, the file server should have the X-Forwarded-For header set and then I would like to include this header in the httpInclude call (or all, I don’t think filtering is necessary)

But I would also understand if this considered beyond the scope of the module, as it might lead to more feature creep along the way.

For me, it would be ‘the missing piece’ to make httpInclude a viable alternative to a another template render engine that has some “advanced” logic that makes use of the header data.

1 Like
8

Oh, wait – we do actually clone the original request header and use it when crafting the request for httpInclude.

I am surprised the header is being dropped? On closer look, I don’t actually see that header in the request logs posted above. Are you sure it exists on the original request?

9

I think I might be missing something in the way Caddy works/adds headers.

What I understand from francislavoie is that caddy won’t include/generate the X-Forwarded-For header if the request doesn’t pass through a reverse proxy.

So this would mean that my initial Caddyfile cannot work:

{
	debug
}
http://localhost:8088 {
	log
	root * web
	handle_path /api/* {
		reverse_proxy localhost:3000
	}
	templates
	file_server
}

Since the page is loaded by:

  1. Access (curl) http://localhost:8088
  2. Which accesses the index.html file served by the file server.
  3. This index file includes an httpInclude "/api/" call that I initially expected to have the X-Forwarded-For header set by Caddy (or the remote_ip)

I don’t know if the file_server sets any headers at all or if any headers are expected to be set by calling the reverse_proxy from the httpInclude call from within the file_server.
This is the point where I am unsure about how to solve it.

I also thought that it might be possible to first pass through a reverse proxy that then forwards everything to the file_server… And I’ve done some additional testing to provide another minimal example but I think I have narrowed down my problem to the special treatment of X-Forwarded-For…
You are correct, the “other” headers are copied by httpInclude. :man_facepalming:

Here is the my new Caddyfile:

{
	debug
	servers http://localhost:8088 {
		trusted_proxies static private_ranges
	}
}

# curl this end-point
# as it passes through a revere-proxy
# I expect it to keep `X-Forwarded-For`
http://localhost:8089 {
	reverse_proxy http://localhost:8088 {
		header_up Custom-Header "my-value"
	}
}

# This is the "hosting" end-point
# that renders my templates and dynamically
# loads content via httpInclude
# from an API that 'parses'
# the headers, i.e. returns `X-Forwarded-For`
http://localhost:8088 {
	log
	root * web
	# according to the debug logs, this request
	# is done without the X-Forwarded-For header
	handle_path /api/* {
		reverse_proxy localhost:3000
	}
	templates
	file_server
}

Using this caddyfile my API does report the Custom-Header which is set by the first reverse proxy.
But I still cannot access the X-Forwarded-For field.
Looking at the documentation I would hope that by setting trusted_proxies static private_ranges for my internal reverse_proxy would allow me keep the header when calling my internal API via httpInclude.
But according to the logs:

2023/09/14 19:39:32.880 INFO    using adjacent Caddyfile
2023/09/14 19:39:32.882 INFO    admin   admin endpoint started  {"address": "localhost:8999", "enforce_origin": false, "origins": ["//localhost:8999", "//[::1]:8999", "//127.0.0.1:8999"]}
2023/09/14 19:39:32.882 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{}]}}, "http": {"servers":{"srv0":{"listen":[":8088"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"web"},{"handler":"templates"}]},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rewrite","strip_path_prefix":"/api"}]},{"handle":[{"handler":"reverse_proxy","upstreams":[{"dial":"localhost:3000"}]}]}]}],"match":[{"path":["/api/*"]}]},{"handle":[{"handler":"file_server","hide":["./Caddyfile"]}]}]}],"terminal":true}],"automatic_https":{"skip":["localhost"]},"logs":{"logger_names":{"localhost:8088":""}}},"srv1":{"listen":[":8089"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Custom-Header":["my-value"]}}},"upstreams":[{"dial":"localhost:8088"}]}]}]}],"terminal":true}],"automatic_https":{"skip":["localhost"]}}}}}
2023/09/14 19:39:32.882 INFO    tls.cache.maintenance   started background certificate maintenance     {"cache": "0xc000408a00"}
2023/09/14 19:39:32.883 DEBUG   http    starting server loop    {"address": "[::]:8088", "tls": false, "http3": false}
2023/09/14 19:39:32.883 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/09/14 19:39:32.883 DEBUG   http    starting server loop    {"address": "[::]:8089", "tls": false, "http3": false}
2023/09/14 19:39:32.883 INFO    http.log        server running  {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2023/09/14 19:39:32.883 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/kai/.local/share/caddy"}
2023/09/14 19:39:32.883 INFO    tls     finished cleaning storage units
2023/09/14 19:39:32.883 INFO    autosaved config (load with --resume flag)      {"file": "/home/kai/.config/caddy/autosave.json"}
2023/09/14 19:39:32.883 INFO    serving initial configuration
2023/09/14 19:39:35.541 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:8088", "total_upstreams": 1}
2023/09/14 19:39:35.542 DEBUG   http.handlers.file_server       sanitized path join     {"site_root": "web", "request_path": "/", "result": "web"}
2023/09/14 19:39:35.542 DEBUG   http.handlers.file_server       located index file      {"filename": "web/index.html"}
2023/09/14 19:39:35.542 DEBUG   http.handlers.file_server       opening file    {"filename": "web/index.html"}
2023/09/14 19:39:35.546 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "", "headers": {"X-Forwarded-Host": ["localhost:8089"], "X-Forwarded-Proto": ["http"], "Accept-Encoding": ["identity"], "User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Custom-Header": ["my-value"], "Caddy-Templates-Include": ["1"], "X-Forwarded-For": ["::1"]}}, "method": "GET", "uri": "/rpc/client"}
2023/09/14 19:39:35.546 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:3000", "total_upstreams": 1}
2023/09/14 19:39:35.550 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:3000", "duration": 0.003286444, "request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "/rpc/client", "headers": {"Caddy-Templates-Include": ["1"], "Accept-Encoding": ["identity"], "User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Custom-Header": ["my-value"]}}, "headers": {"Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8"], "Date": ["Thu, 14 Sep 2023 19:39:35 GMT"], "Server": ["postgrest/11.2.0"]}, "status": 200}
2023/09/14 19:39:35.550 INFO    http.log.access handled request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "", "headers": {"Accept": ["*/*"], "Custom-Header": ["my-value"], "Caddy-Templates-Include": ["1"], "X-Forwarded-For": ["::1"], "X-Forwarded-Host": ["localhost:8089"], "X-Forwarded-Proto": ["http"], "Accept-Encoding": ["identity"], "User-Agent": ["curl/8.2.1"]}}, "bytes_read": 0, "user_id": "", "duration": 0.003661115, "size": 194, "status": 200, "resp_headers": {"Content-Type": ["application/json; charset=utf-8"], "Server": ["Caddy", "postgrest/11.2.0"], "Date": ["Thu, 14 Sep 2023 19:39:35 GMT"], "Content-Range": ["0-0/*"]}}
2023/09/14 19:39:35.550 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "", "headers": {"User-Agent": ["curl/8.2.1"], "Caddy-Templates-Include": ["1"], "Accept": ["*/*"], "Custom-Header": ["my-value"], "X-Forwarded-For": ["::1"], "X-Forwarded-Host": ["localhost:8089"], "X-Forwarded-Proto": ["http"], "Accept-Encoding": ["identity"]}}, "method": "GET", "uri": "/rpc/sample"}
2023/09/14 19:39:35.550 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:3000", "total_upstreams": 1}
2023/09/14 19:39:35.551 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:3000", "duration": 0.000734238, "request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "/rpc/sample", "headers": {"Caddy-Templates-Include": ["1"], "Accept": ["*/*"], "Custom-Header": ["my-value"], "Accept-Encoding": ["identity"], "User-Agent": ["curl/8.2.1"]}}, "headers": {"Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8", "text/html; charset=utf-8"], "Date": ["Thu, 14 Sep 2023 19:39:35 GMT"], "Server": ["postgrest/11.2.0"]}, "status": 200}
2023/09/14 19:39:35.551 INFO    http.log.access handled request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "", "headers": {"X-Forwarded-For": ["::1"], "X-Forwarded-Host": ["localhost:8089"], "X-Forwarded-Proto": ["http"], "Accept-Encoding": ["identity"], "User-Agent": ["curl/8.2.1"], "Caddy-Templates-Include": ["1"], "Accept": ["*/*"], "Custom-Header": ["my-value"]}}, "bytes_read": 0, "user_id": "", "duration": 0.000852541, "size": 280, "status": 200, "resp_headers": {"Server": ["Caddy", "postgrest/11.2.0"], "Date": ["Thu, 14 Sep 2023 19:39:35 GMT"], "Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8", "text/html; charset=utf-8"]}}
2023/09/14 19:39:35.551 INFO    http.log.access handled request {"request": {"remote_ip": "::1", "remote_port": "59056", "client_ip": "::1", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "/", "headers": {"X-Forwarded-For": ["::1"], "X-Forwarded-Host": ["localhost:8089"], "X-Forwarded-Proto": ["http"], "Accept-Encoding": ["gzip"], "User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Custom-Header": ["my-value"]}}, "bytes_read": 0, "user_id": "", "duration": 0.009296267, "size": 1319, "status": 200, "resp_headers": {"Content-Type": ["text/html; charset=utf-8"], "Content-Length": ["1319"], "Server": ["Caddy"]}}
2023/09/14 19:39:35.551 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:8088", "duration": 0.010346408, "request": {"remote_ip": "::1", "remote_port": "38770", "client_ip": "::1", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "/", "headers": {"X-Forwarded-For": ["::1"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Host": ["localhost:8089"], "User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Custom-Header": ["my-value"]}}, "headers": {"Content-Length": ["1319"], "Content-Type": ["text/html; charset=utf-8"], "Server": ["Caddy"], "Date": ["Thu, 14 Sep 2023 19:39:35 GMT"]}, "status": 200}
^C2023/09/14 19:39:39.800       INFO    shutting down   {"signal": "SIGINT"}
2023/09/14 19:39:39.800 WARN    exiting; byeee!! 👋     {"signal": "SIGINT"}
2023/09/14 19:39:39.800 INFO    http    servers shutting down with eternal grace period
2023/09/14 19:39:39.801 INFO    admin   stopped previous server {"address": "localhost:8999"}
2023/09/14 19:39:39.801 INFO    shutdown complete       {"signal": "SIGINT", "exit_code": 0}

Specifically:

2023/09/14 19:39:35.551 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:3000", "duration": 0.000734238, "request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8089", "uri": "/rpc/sample", "headers": {"Caddy-Templates-Include": ["1"], "Accept": ["*/*"], "Custom-Header": ["my-value"], "Accept-Encoding": ["identity"], "User-Agent": ["curl/8.2.1"]}}, "headers": {"Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8", "text/html; charset=utf-8"], "Date": ["Thu, 14 Sep 2023 19:39:35 GMT"], "Server": ["postgrest/11.2.0"]}, "status": 200}

Shows that the X-Forwarded-For header is dropped.

I bet I am missing something obvious/basic…

EDIT: The X-Forwarded-For header is dropped when doing the httpInclude request but does exist in the reverse proxy step before that

10

So with a bit more playing around, I was able to fix it by doing this weird little trick:

http://localhost:8088 {
	log
	root * web
	# according to the debug logs, this request
	# is done without the X-Forwarded-For header
	handle_path /api/* {
		reverse_proxy localhost:3000 {
                        # Manually setting it seems to fix it.
			header_up X-Forwarded-For {header.X-Forwarded-For}
		}
	}
	templates
	file_server
}

By manually setting the X-Forwarded-For header in the internal reverse_proxy it does stick around for the request and my API can access the field.

Though I do not really understand why this works and if even if this should work to begin with…
I would appreciate any comments to clarify if this is a solution that others could use. :sweat_smile:

1 Like
11

Ah - it’s because you didn’t configure trusted_proxies. You should do that.

1 Like
12

Ah sorry, maybe I should’ve posted the entire caddy file again.
If you look at my previous post I did include the trusted proxies (I think I configured it correctly?)

But I guess it sounds like I did make a mistake when configuring them

1 Like
13

Ah yeah, remove the http://localhost:8088 there. Servers takes a listener address, not a proxy upstream address. Omitting it applies it to all servers.

1 Like
14

Still doesn’t work…
Here is the entire Caddyfile again:

{
	debug
	servers {
		trusted_proxies static private_ranges
	}
}

# curl this end-point
# as it passes through a revere-proxy
# I expect it to keep `X-Forwarded-For`
http://localhost:8089 {
	reverse_proxy http://localhost:8088 {
		header_up Custom "my-value"
	}
}

# This is the "hosting" end-point
# that renders my templates and dynamically
# loads content via httpInclude
# from an API that 'parses'
# the headers, i.e. returns `X-Forwarded-For`
http://localhost:8088 {
	log
	root * web
	# according to the debug logs, this request
	# is done without the X-Forwarded-For header
	handle_path /api/* {
		reverse_proxy localhost:3000 {
			header_up CIP {header.Custom}
			#header_up X-Forwarded-For {header.X-Forwarded-For}
		}
	}
	templates
	file_server
}

Hm, but wasn’t it already the correct listener address or should it’ve been just :8088 ?
But also setting it globally doesn’t change anything.

Here the log output if it helps:

2023/09/15 06:54:24.739 INFO    using adjacent Caddyfile
2023/09/15 06:54:24.741 INFO    admin   admin endpoint started  {"address": "localhost:8999", "enforce_origin": false, "origins": ["//localhost:8999", "//[::1]:8999", "//127.0.0.1:8999"]}
2023/09/15 06:54:24.742 INFO    tls.cache.maintenance   started background certificate maintenance     {"cache": "0xc000257b80"}
2023/09/15 06:54:24.742 DEBUG   http.auto_https adjusted config {"tls": {"automation":{"policies":[{}]}}, "http": {"servers":{"srv0":{"listen":[":8088"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"vars","root":"web"},{"handler":"templates"}]},{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"rewrite","strip_path_prefix":"/api"}]},{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Cip":["{http.request.header.Custom}"]}}},"upstreams":[{"dial":"localhost:3000"}]}]}]}],"match":[{"path":["/api/*"]}]},{"handle":[{"handler":"file_server","hide":["./Caddyfile"]}]}]}],"terminal":true}],"automatic_https":{"skip":["localhost"]},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"},"logs":{"logger_names":{"localhost:8088":""}}},"srv1":{"listen":[":8089"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"Custom":["my-value"]}}},"upstreams":[{"dial":"localhost:8088"}]}]}]}],"terminal":true}],"automatic_https":{"skip":["localhost"]},"trusted_proxies":{"ranges":["192.168.0.0/16","172.16.0.0/12","10.0.0.0/8","127.0.0.1/8","fd00::/8","::1"],"source":"static"}}}}}
2023/09/15 06:54:24.743 DEBUG   http    starting server loop    {"address": "[::]:8088", "tls": false, "http3": false}
2023/09/15 06:54:24.743 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/09/15 06:54:24.743 DEBUG   http    starting server loop    {"address": "[::]:8089", "tls": false, "http3": false}
2023/09/15 06:54:24.743 INFO    http.log        server running  {"name": "srv1", "protocols": ["h1", "h2", "h3"]}
2023/09/15 06:54:24.743 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/kai/.local/share/caddy"}
2023/09/15 06:54:24.743 INFO    tls     finished cleaning storage units
2023/09/15 06:54:24.743 INFO    autosaved config (load with --resume flag)      {"file": "/home/kai/.config/caddy/autosave.json"}
2023/09/15 06:54:24.743 INFO    serving initial configuration
2023/09/15 06:54:26.377 DEBUG   http.handlers.file_server       sanitized path join     {"site_root": "web", "request_path": "/", "result": "web"}
2023/09/15 06:54:26.377 DEBUG   http.handlers.file_server       located index file      {"filename": "web/index.html"}
2023/09/15 06:54:26.377 DEBUG   http.handlers.file_server       opening file    {"filename": "web/index.html"}
2023/09/15 06:54:26.382 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"Accept": ["*/*"], "User-Agent": ["curl/8.2.1"], "Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"]}}, "method": "GET", "uri": "/rpc/client"}
2023/09/15 06:54:26.382 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:3000", "total_upstreams": 1}
2023/09/15 06:54:26.389 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:3000", "duration": 0.006864586, "request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "/rpc/client", "headers": {"Cip": [""], "Caddy-Templates-Include": ["1"], "Accept": ["*/*"], "User-Agent": ["curl/8.2.1"], "Accept-Encoding": ["identity"]}}, "headers": {"Date": ["Fri, 15 Sep 2023 06:54:26 GMT"], "Server": ["postgrest/11.2.0"], "Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8"]}, "status": 200}
2023/09/15 06:54:26.389 INFO    http.log.access handled request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"Caddy-Templates-Include": ["1"], "Accept": ["*/*"], "User-Agent": ["curl/8.2.1"], "Accept-Encoding": ["identity"]}}, "bytes_read": 0, "user_id": "", "duration": 0.00710689, "size": 176, "status": 200, "resp_headers": {"Server": ["Caddy", "postgrest/11.2.0"], "Date": ["Fri, 15 Sep 2023 06:54:26 GMT"], "Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8"]}}
2023/09/15 06:54:26.389 DEBUG   http.handlers.rewrite   rewrote request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"]}}, "method": "GET", "uri": "/rpc/sample"}
2023/09/15 06:54:26.389 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "localhost:3000", "total_upstreams": 1}
2023/09/15 06:54:26.390 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "localhost:3000", "duration": 0.001068055, "request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "/rpc/sample", "headers": {"Accept-Encoding": ["identity"], "Cip": [""], "Caddy-Templates-Include": ["1"], "User-Agent": ["curl/8.2.1"], "Accept": ["*/*"]}}, "headers": {"Server": ["postgrest/11.2.0"], "Content-Range": ["0-0/*"], "Content-Type": ["application/json; charset=utf-8", "text/html; charset=utf-8"], "Date": ["Fri, 15 Sep 2023 06:54:26 GMT"]}, "status": 200}
2023/09/15 06:54:26.390 INFO    http.log.access handled request {"request": {"remote_ip": "", "remote_port": "", "client_ip": "", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "", "headers": {"User-Agent": ["curl/8.2.1"], "Accept": ["*/*"], "Accept-Encoding": ["identity"], "Caddy-Templates-Include": ["1"]}}, "bytes_read": 0, "user_id": "", "duration": 0.001251965, "size": 280, "status": 200, "resp_headers": {"Content-Type": ["application/json; charset=utf-8", "text/html; charset=utf-8"], "Server": ["Caddy", "postgrest/11.2.0"], "Date": ["Fri, 15 Sep 2023 06:54:26 GMT"], "Content-Range": ["0-0/*"]}}
2023/09/15 06:54:26.391 INFO    http.log.access handled request {"request": {"remote_ip": "::1", "remote_port": "55046", "client_ip": "::1", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8088", "uri": "/", "headers": {"User-Agent": ["curl/8.2.1"], "Accept": ["*/*"]}}, "bytes_read": 0, "user_id": "", "duration": 0.013189619, "size": 1301, "status": 200, "resp_headers": {"Content-Length": ["1301"], "Server": ["Caddy"], "Content-Type": ["text/html; charset=utf-8"]}}
^C2023/09/15 06:54:33.240       INFO    shutting down   {"signal": "SIGINT"}
2023/09/15 06:54:33.240 WARN    exiting; byeee!! 👋     {"signal": "SIGINT"}
2023/09/15 06:54:33.240 INFO    http    servers shutting down with eternal grace period
2023/09/15 06:54:33.241 INFO    admin   stopped previous server {"address": "localhost:8999"}
2023/09/15 06:54:33.241 INFO    shutdown complete       {"signal": "SIGINT", "exit_code": 0}

But thanks again for the help until now. I guess we are close :slight_smile:

15

What the heck, why is remote_ip empty? :thinking: That must be the problem. I guess httpInclude makes the request with an empty remote address, somehow…

16

Correct, because there is no socket and thus no connection. It’s very lightweight :smiley:

17

But shouldn’t the header X-Forwarded-For be copied as well?
Either way, I guess the the ‘header_up’ trick is okay…ish?

18

Be copied from where to where?

19

This seems to work.
But I thought that this would not be required if the server is trusted (and therefore the x-forwarded-for header is passed through the internal reverse proxy.

20

Like I said, the problem is that remote_ip is empty so the logic for XFF gets confused and does nothing, because there’s no way to safely handle the header when the remote address is empty. It’s not possible to add the directly connecting client’s IP, so the chain of IPs would be invalid.

@matt can we just set RemoteAddr to something like ::1 on httpInclude requests? That would fix it.