Is Caddy mutating header content from logging settings?

Help
1

1. The problem I’m having:

I’m attempting to hash sensitive headers in access logs via logging settings. Per the documentation, I should be able to change the format and hash them, which - while it works - appears to also be changing the headers to their hashed values to reverse proxy endpoints?

2. Error messages and/or full log output:

Here’s the example Caddyfile:

:80 {
	log {
		format filter {
			wrap json
			fields {
				request>headers>X-Test hash
			}
		}
	}

	reverse_proxy http://127.0.0.1:1337
}

Here’s my reverse proxy listener:

$ nc -l 1337

And here’s my request:

$ curl localhost -H'X-Test:"test one two three"'

Here’s what my caddy run logs (after I Ctrl-C the simple netcat listener):

{"level":"error","ts":1693016089.716869,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_ip":"127.0.0.1","remote_port":"43332","client_ip":"127.0.0.1","proto":"HTTP/1.1","method":"GET","host":"localhost","uri":"/","headers":{"User-Agent":["curl/8.1.1"],"Accept":["*/*"],"X-Test":["338145b3"]}},"bytes_read":0,"user_id":"","duration":0.000283437,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}

And here’s what the reverse proxied endpoint (netcat) sees:

GET / HTTP/1.1
Host: localhost
User-Agent: curl/8.1.1
Accept: */*
X-Forwarded-For: 127.0.0.1
X-Forwarded-Host: localhost
X-Forwarded-Proto: http
X-Test: 338145b3
Accept-Encoding: gzip

As you can see I don’t get test one two three from Caddy but the hashed value instead as it appears in the logs, as if the header itself is being mutated, not only at log time.

3. Caddy version:

I grabbed the latest stable release from GitHub to reproduce the issue.

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

Just nabbed a GitHub release.

a. System environment:

I’m on NixOS but the caddy I got was statically linked so I don’t think there’s anything unique here.

b. Command:

./caddy run

c. Service/unit/compose file:

I just ran it manually.

d. My complete Caddy config:

:80 {
	log {
		format filter {
			wrap json
			fields {
				request>headers>X-Test hash
			}
		}
	}

	reverse_proxy http://127.0.0.1:1337
}

5. Links to relevant resources:

I’m following the log documentation, but maybe I’m misunderstanding how this works? In any event I’m surprised to find the header altered just from making logging changes.

2

Update: it appears that this is only happening with the hash action - using directives like delete work as expected.

3

You’re right, that’s a bug. Go is pretty tricky. We thought we were modifying a copy of the headers but it’s not a copy :grimacing: So we’ll need to copy the header values when transforming them which has a tiny performance impact from making an allocation but :man_shrugging: it’s what we gotta do I think.

4

Thank you @francislavoie, I appreciate the triage and fix :pray:

1 Like
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.