Can't use copy_headers in clear-text forward_auth to pass to reverse_proxy upstream

TuggyNE · August 21, 2024, 8:54pm

1. The problem I’m having:

I’m using Caddy as a top-level reverse proxy for several microservices, including one for auth via forward_auth, which returns a custom header called X-User-Facts. This is supposed to be passed to the appropriate other microservice via reverse_proxy and copied back to the final response. Copying to the response works reliably. Unfortunately, the upstream microservices only see the literal Caddy placeholder for the header name, not the value of it: log messages contain the literal string {http.reverse_proxy.header.X-User-Facts} rather than any actual header value.

This is all being run on a single dev machine, with /etc/hosts including some custom domains for TLS support, and Cloudflare DNS-01:

127.0.0.1	iops3-dev.farmersfriendllc.com
127.0.0.1	auth.iops3-dev.farmersfriendllc.com

Synthesizing fake auth and upstream microservices with Caddy listeners produces an interesting result: I get the correct behavior if the mock for forward_auth uses TLS in its HTTP transport, but not if it is in cleartext. (It doesn’t matter whether reverse_proxy is using TLS or not.)

Obvious workaround having gone through all this

Sticking another Caddy layer around the real auth microservice to enable forward_auth via TLS is effective, but it’s an annoying hack and I don’t understand why it’s necessary. (It’s also a trap for the unwary, since one must use header_up Host {upstream_hostport} or similar rewrite to avoid infinite loops. I suppose it’s also a bit of a performance drain, but a few ms latency and a small extra CPU usage are unlikely to be too big a deal in our case.)

2. Error messages and/or full log output:

After curl -L https://iops3-dev.farmersfriendllc.com:8585/inv/parts:

{"level":"debug","ts":"2024-08-21T15:33:27.644-0500","logger":"http.handlers.reverse_proxy","msg":"selected upstream","dial":"localhost:9004","total_upstreams":1}
{"level":"debug","ts":"2024-08-21T15:33:27.645-0500","logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"localhost:9004","duration":0.000805378,"request":{"remote_ip":"127.0.0.1","remote_port":"63399","client_ip":"127.0.0.1","proto":"HTTP/2.0","method":"GET","host":"iops3-dev.farmersfriendllc.com:8585","uri":"/","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["iops3-dev.farmersfriendllc.com:8585"],"X-Forwarded-Method":["GET"],"X-Forwarded-Uri":["/inv/parts"],"User-Agent":["curl/8.4.0"],"Accept":["*/*"],"X-Forwarded-For":["127.0.0.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"iops3-dev.farmersfriendllc.com"}},"headers":{"Server":["Caddy"],"Date":["Wed, 21 Aug 2024 20:33:27 GMT"],"Content-Length":["0"]},"status":200}
{"level":"debug","ts":"2024-08-21T15:33:27.645-0500","logger":"http.handlers.reverse_proxy","msg":"handling response","upstream":"localhost:9004","duration":0.000805378,"request":{"remote_ip":"127.0.0.1","remote_port":"63399","client_ip":"127.0.0.1","proto":"HTTP/2.0","method":"GET","host":"iops3-dev.farmersfriendllc.com:8585","uri":"/","headers":{"X-Forwarded-Proto":["https"],"X-Forwarded-Host":["iops3-dev.farmersfriendllc.com:8585"],"X-Forwarded-Method":["GET"],"X-Forwarded-Uri":["/inv/parts"],"User-Agent":["curl/8.4.0"],"Accept":["*/*"],"X-Forwarded-For":["127.0.0.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"iops3-dev.farmersfriendllc.com"}},"handler":0}
{"level":"debug","ts":"2024-08-21T15:33:27.645-0500","logger":"http.handlers.rewrite","msg":"rewrote request","request":{"remote_ip":"127.0.0.1","remote_port":"63399","client_ip":"127.0.0.1","proto":"HTTP/2.0","method":"GET","host":"iops3-dev.farmersfriendllc.com:8585","uri":"/inv/parts","headers":{"User-Agent":["curl/8.4.0"],"Accept":["*/*"],"X-User-Facts":["{http.reverse_proxy.header.X-User-Facts}"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"iops3-dev.farmersfriendllc.com"}},"method":"GET","uri":"/parts"}

You can see the uninterpreted placeholder in the headers on the last line, and no sign of the X-User-Facts header otherwise.

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

xcaddy build --with github.com/caddy-dns/cloudflare and then copied the binary to an appropriate place.

a. System environment:

macOS 12.7.5 (Intel).

b. Command:

caddy start --watch

(For most of these, I also had to add --adapter caddyfile --config <xyz>.Caddyfile.)

c. Service/unit/compose file:

(None.)

d. My complete Caddy config:

Original, suitably trimmed to give minimal repro case:

{
	#email ... whatever...
	acme_dns cloudflare {env.CF_API_TOKEN}
	log default {
		output file caddy_log/json.log
		format json {
			time_format iso8601
		}
	}
	debug
}

iops3-dev.farmersfriendllc.com:8585 {
	# XXX: Non-functional, but https://iops3-dev.farmersfriendllc.com:9003 is fine
	forward_auth localhost:9004 {
		uri /
		copy_headers X-User-Facts
	}

	handle_path /inv/* {
		reverse_proxy https://iops3-dev.farmersfriendllc.com:9001 {
		}
	}

	log

	encode zstd gzip
}

Fake upstream (also tried adding http://iops3-dev.farmersfriendllc.com:9000 block and adjusting others to reference it, but it made no difference):

{
	#email ... whatever...
	acme_dns cloudflare {env.CF_API_TOKEN}
	log default {
		output file caddy_log/test.log
		format json {
			time_format iso8601
		}
	}
	debug
}

iops3-dev.farmersfriendllc.com:9001 {
	respond "X-User-Facts was: {header.X-User-Facts}"
	handle_errors {
		respond "{err.status_code} (pseudo-micro-service) {err.status_text}"
	}

	log
}

Fake auth:

{
	#email ... whatever...
	acme_dns cloudflare {env.CF_API_TOKEN}
	log default {
		output file caddy_log/test-auth.log
		format json {
			time_format iso8601
		}
	}
	debug
}

iops3-dev.farmersfriendllc.com:9003 {
	header X-User-Facts "Testing value"
	respond 200 {
		body "OK"
	}
	handle_errors {
		respond "{err.status_code} (pseudo-auth-service) {err.status_text}"
	}

	log
}

http://localhost:9004 {
	header X-User-Facts "Testing value"
	respond 200 {
		body "OK"
	}
	handle_errors {
		respond "{err.status_code} (pseudo-auth-service) {err.status_text}"
	}

	log
}

5. Links to relevant resources:

Composing in the Caddyfile seemed potentially relevant, especially given Forwarding User Auth ID as a HTTP header and when early testing seemed to show that it made a difference whether I had one or several Caddyfiles and processes, but when I isolated it down to whether forward_auth was using TLS or not, it didn’t seem like it was relevant anymore.
Forward_auth copy_headers value not replaced also gave me some hassle initially as I was relying on the web standards and using a non-canonical casing, but changing that across the board still left this problem.
Forward_auth overriding reverse_proxy directive turned out to be third-party software.

francislavoie · August 22, 2024, 5:34am

Your forward_auth backend doesn’t actually reply with X-User-Facts so there’s no header value, so the placeholder doesn’t get replaced.

The problem is that your upstream uses localhost as the Host matcher, but forward_auth (and reverse_proxy) preserve the original Host header, so your upstream server doesn’t match the site block your defined and instead simply responds with an empty response.

Either remove localhost from your site address (i.e. just use :9004 as the site address) or override the Host header with header_up Host localhost in your forward_auth.

system · September 21, 2024, 5:35am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.