Header filters - case sensitive?

aaroncarson · February 6, 2023, 12:36am

1. Caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I installed, and run Caddy:

a. System environment:

Linux Caddy 5.15.83-1-pve #1 SMP PVE 5.15.83-1 (2022-12-15T00:00Z) x86_64 Linux
Alpine Linux 3.16
OpenRC [LXC] 0.44.10
In a Proxmox LXC

b. Command:

caddy start --config /etc/caddy/Caddyfile --pidfile /etc/caddy/caddy.pid

c. Service/unit/compose file:

#!/sbin/openrc-run

description="Caddy web server"
description_reload="Reload configuration"

extra_started_commands="reload"

: ${CADDY_DIR:=/etc/caddy}
: ${CADDYFILE:=/etc/caddy/Caddyfile}
: ${CADDY_PIDFILE:=/etc/caddy/caddy.pid}
: ${CADDY_USER:=caddy}

depend() {
        need net
        need localmount
        use dns
        after firewall
        provide webserver
}

start() {
	export HOME=${CADDY_DIR}
        ebegin "Starting caddy"
	caddy start --config ${CADDYFILE} --pidfile ${CADDY_PIDFILE}
        eend $?
}

stop() {
	export HOME=${CADDY_DIR}
        ebegin "Stopping caddy"
	caddy stop
        eend $?
}

reload() {
	export HOME=${CADDY_DIR}
        ebegin "Reloading caddy configuration"
	caddy reload --config ${CADDYFILE} --force
        eend $?
}

d. My complete Caddy config:

# Snippets

(cloudflare) {
	tls {
		dns cloudflare [REDACTED]
		resolvers 1.1.1.1
	}
}

(rev_untrusted_https) {
	transport http {
		tls_insecure_skip_verify
	}
}

(access_only_local) {
	@denied_public not remote_ip private_ranges
	abort @denied_public
}

(access_only_me) {
	@denied_not_me not remote_ip [REDACTED] private_ranges
	abort @denied_not_me
}

(log_to_file) {
	log {
		output file /var/log/caddy/{args.0}.log {
			roll_uncompressed
		}
		format filter {
			wrap json
			fields {
				common_log delete
				request>headers>authorization delete
				request>headers>X-Egg-Inc-Id replace EI_redacted
				request>remote_ip ip_mask {
					ipv4 24
					ipv6 56
				}
				request>headers>Cf-Connecting-Ip ip_mask {
					ipv4 24
					ipv6 56
				}
				request>headers>X-Forwarded-For ip_mask {
					ipv4 24
					ipv6 56
				}
			}
		}
	}
}

# Globals
{
	email [REDACTED]
	acme_ca "https://dv.acme-v02.api.pki.goog/directory"
	acme_eab {
		key_id [REDACTED]
		mac_key [REDACTED]
	}
	skip_install_trust
	log {
		format json
		output file /var/log/caddy/stdout.log {
			roll_uncompressed
		}
		exclude http.log.access
	}
	servers {
		metrics
	}
}

# Sites
api.adhd.energy {
	handle_path /egg_inc/* {
		reverse_proxy http://192.168.6.18:5648
	}

	import log_to_file api.adhd.energy
}

3. The problem I’m having:

Hi! I’m using Caddy as an API proxy for a project I’m working on (GitHub - caraar12345/ei-api: Egg, Inc. API) and part of that requires including a header X-Egg-Inc-Id which is very loosely comparable to an API key. So I wouldn’t like to log it for obvious reasons.

However, I’m finding that it’s case sensitive. Setting the Caddyfile to filter on x-egg-inc-id doesn’t do anything if make the request with X-Egg-Inc-Id.

My concern here: if the filter is case sensitive, it’s likely that people are inadvertently logging Authorization headers too.

Is the issue with me, my config, or Caddy?

4. Error messages and/or full log output:

❯ http GET https://api.adhd.energy/egg_inc/epic_research/calculator/json X-Egg-Inc-ID:[REDACTED]
HTTP/1.1 200 OK
Alt-Svc: h3=":443"; ma=2592000
Content-Length: 1178
Content-Type: application/json
Date: Mon, 06 Feb 2023 00:08:56 GMT
Server: Caddy, uvicorn

Case matches

{
  "msg": "handled request",
  "request": {
    "headers": {
      "X-Egg-Inc-Id": "EI_redacted"
      "Accept": [
        "*/*"
      ],
      "Connection": [
        "keep-alive"
      ],
      "User-Agent": [
        "HTTPie/3.2.1"
      ],
      "Accept-Encoding": [
        "gzip, deflate"
      ]
    },
    "remote_ip": "0.0.0.0",
    "method": "GET",
    "proto": "HTTP/1.1",
    "host": "api.adhd.energy",
    "remote_port": "55139",
    "tls": {
      "server_name": "api.adhd.energy",
      "cipher_suite": 4865,
      "proto": "http/1.1",
      "resumed": false,
      "version": 772
    },
    "uri": "/egg_inc/epic_research/calculator/json"
  },
  "agent": {
    "name": "Caddy",
    "id": "ad77225f-b43f-401d-afd7-874a901bb6aa",
    "type": "filebeat",
    "ephemeral_id": "3206da8b-e5ff-4297-8dee-bc427eb1745d",
    "version": "8.5.0"
  },
  "log": {
    "file": {
      "path": "/var/log/caddy/api.adhd.energy.log"
    },
    "offset": 3611
  },
  "level": "info",
  "logger": "http.log.access.log7",
  "resp_headers": {
    "Server": [
      "Caddy",
      "uvicorn"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ],
    "Content-Length": [
      "1178"
    ],
    "Date": [
      "Mon, 06 Feb 2023 00:08:14 GMT"
    ],
    "Content-Type": [
      "application/json"
    ]
  },
  "duration": 0.476357181,
  "input": {
    "type": "filestream"
  },
  "@timestamp": "2023-02-06T00:08:15.557Z",
  "ecs": {
    "version": "8.0.0"
  },
  "size": 1178,
  "user_id": "",
  "status": 200,
  "ts": 1675642095.5577335
}

Case doesn’t match

{
  "msg": "handled request",
  "request": {
    "headers": {
      "X-Egg-Inc-Id": [
        "[REDACTED]"
      ],
      "Accept": [
        "*/*"
      ],
      "Connection": [
        "keep-alive"
      ],
      "User-Agent": [
        "HTTPie/3.2.1"
      ],
      "Accept-Encoding": [
        "gzip, deflate"
      ]
    },
    "remote_ip": "0.0.0.0",
    "method": "GET",
    "proto": "HTTP/1.1",
    "host": "api.adhd.energy",
    "remote_port": "55139",
    "tls": {
      "server_name": "api.adhd.energy",
      "cipher_suite": 4865,
      "proto": "http/1.1",
      "resumed": false,
      "version": 772
    },
    "uri": "/egg_inc/epic_research/calculator/json"
  },
  "agent": {
    "name": "Caddy",
    "id": "ad77225f-b43f-401d-afd7-874a901bb6aa",
    "type": "filebeat",
    "ephemeral_id": "3206da8b-e5ff-4297-8dee-bc427eb1745d",
    "version": "8.5.0"
  },
  "log": {
    "file": {
      "path": "/var/log/caddy/api.adhd.energy.log"
    },
    "offset": 3611
  },
  "level": "info",
  "logger": "http.log.access.log7",
  "resp_headers": {
    "Server": [
      "Caddy",
      "uvicorn"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ],
    "Content-Length": [
      "1178"
    ],
    "Date": [
      "Mon, 06 Feb 2023 00:08:14 GMT"
    ],
    "Content-Type": [
      "application/json"
    ]
  },
  "duration": 0.476357181,
  "input": {
    "type": "filestream"
  },
  "@timestamp": "2023-02-06T00:08:15.557Z",
  "ecs": {
    "version": "8.0.0"
  },
  "size": 1178,
  "user_id": "",
  "status": 200,
  "ts": 1675642095.5577335
}

5. What I already tried:

I’ve tried various capitalisations, filters (regexp, replace…) - to no avail.
Any ideas would really help please! Thanks

francislavoie · February 6, 2023, 1:37am

Caddy canonicalizes header fields. That means it’s in Upper-Kebab-Case. This is done automatically by the Go stdlib, via the textproto.CanonicalMIMEHeaderKey function which is applied on all header operations.

The issue with log filters is that we don’t make a distinction of the type of data in the logs, so the filter keys need to match exactly what’s in the logs.

The Authorization header values are not logged by default. See the log_credentials server global option for more details on that. The values will be empty, but the field will exist in the logs to show that it existed in the request.

aaroncarson · February 6, 2023, 1:52am

I see, I see!

This makes a lot of sense - thank you!

My only consideration is that perhaps Caddy should default to purely lowercase; particularly with the move to HTTP/2, lowercase is “enforced” in RFC 7540.

But even so, whilst it does canonicalise them the way it does, it would probably be worth making the docs a tad clearer as I’m sure I’m not the first person to have been confused by that one

Thanks for the reply Francis (and thanks for all your work on Caddy in general!!)

francislavoie · February 6, 2023, 2:16am

Lowercase is enforced on the wire for HTTP/2, that’s true, but user-land is a different story. Canonicalized headers are easier to read in general, and it’s what the Go stdlib uses everywhere, so it’s more natural to use that.

FYI, RFC 9110 is the more relevant spec to follow because it speaks to HTTP in general, unspecific to a particular version. Note that registered header fields are all listed in their canonical form.

system · March 8, 2023, 2:17am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.