How to override reverse proxy response headers with path matcher?

1. Caddy version (caddy version):

v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

2. How I run Caddy:

systemd

a. System environment:

centos 8, systemd 239-45.el8_4.3, kernel 4.18.0-305.12.1.el8_4.x86_64

b. Command:

systemctl reload caddy
or
systemctl restart caddy
because reload command often hangs and fails

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/local/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/local/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

gitea.moe {
	import logto gitea.moe
	header /gitea.moe/css/raw/branch/master/custom.css Content-Type "text/css; charset=utf-8"
	reverse_proxy localhost:3400
}

*.gitea.moe {
	tls {
		import dns-cloudflare-token
	}
	@rw {
		header_regexp host Host ^(\w+)\.gitea\.moe$
		path_regexp path ^\/([^\/]+)\/(.*)
	}
	rewrite @rw /{re.host.1}/{re.path.1}/raw/branch/master/{re.path.2}
	header *.html Content-Type "text/html"
	header *.js Content-Type "text/js"
	header *.css Content-Type "text/css"
	reverse_proxy @rw localhost:3400 {
		@nf {
			status 404
		}
		handle_response @nf {
			respond 404
		}
	}
}

3. The problem I’m having:

I want the reverse proxy to override Content-Type headers from upstream, according to file extension at end of URL. I thought it should be as simple as header *.html Content-Type "text/html" but it’s not working at all; there is no change in response headers, with or without {defer}.

The first site block has been working for a while, but response has two Content-Type headers. It works so I don’t care, but thought it’s worth mentioning. I am working on the second block.

How should I rewrite response headers with reverse_proxy? Am I doing it wrong, or am I encountering bugs?

4. Error messages and/or full log output:

I only have this for the issue with one of the things I tried below.

Oct 06 20:23:14 avps.owo69.me systemd[1]: Reloading Caddy.
Oct 06 20:23:14 avps.owo69.me caddy[3605456]: {"level":"info","ts":1633569794.9079423,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter"
:""}
Oct 06 20:23:14 avps.owo69.me caddy[3605456]: reload: adapting config using caddyfile: parsing caddyfile tokens for 'header': /etc/caddy/misc-sites.Caddyfile:45 - Error during
parsing: cannot specify headers in both arguments and block
Oct 06 20:23:14 avps.owo69.me systemd[1]: caddy.service: Control process exited, code=exited status=1
Oct 06 20:23:14 avps.owo69.me systemd[1]: Reload failed for Caddy.

5. What I already tried:

I have tried them with {defer}, which I thought would have been needed. But it makes no difference for the second block, and for the first site block, makes it not work either.

I have also tried the following to delete and then add the header, but I got the error “cannot specify headers in both arguments and block”, which makes no sense because there weren’t any headers in arguments:

I also tried putting the header directives in a route{}.

6. Links to relevant resources:

wat

The key here is in the Syntax section of the request matcher docs:

Path matchers must start with / when used inline. If using a suffix matcher, then you need to use a named matcher instead.

@html path *.html
header @html Content-Type text/html

If you adapt your config to JSON with caddy adapt --pretty, you’ll see what’s actually happening instead; the Caddyfile parser is reading *.html as the header field, then Content-Type as the search for a replacement, and text/html as the replacement value… which doesn’t make sense at all.

Also, you’ll likely need to set defer in this case so that the header adjustments happen after the proxy executes:

But I have to ask, why do you need this at all? Why doesn’t your proxy upstream set the content type correctly? This shouldn’t be something you need to configure Caddy to fix, it should be correctly set by the upstream already. I’d look for a bug there instead.

This usage of handle_response doesn’t really make sense :thinking: if it’s a 404, then you’re responding with a 404. That’s not very useful.

Oh. It’s one small statement that’s easily overlooked and not logically inferrable from everything else that I’ve seen seen, but knowing this has solved my problem so thanks.

Also I found out that the block must have spaces between the braces: {defer} does not work but { defer } does. I figured that out when I tried to put named matchers on one line and got this error. Also wasn’t expected as this is possible with any other language I’ve used.

It’s a hack to provide a github pages-like feature for gitea, which is not my project. I have absolutely no experience with go so modifying it would be a relatively daunting task, and then I’d have to maintain a fork. Save for the trouble of learning the minutial nuances of caddy (helpful to know anyway), this is a simple and easily manageable solution. And it would work great if I can implement caching in caddy, but it’s a fine experiment for now as my site is very small.

The upstream sends a templated 404 html page that’s broken due to the path rewrites.

1 Like

This is all explained in the Caddyfile Concepts page – you should always use newlines for blocks. If it works without newlines, it’s honestly a bug in that case, and is unintended behaviour.

1 Like