Correctly preserve `X-Forwarded-*` headers when Caddy is not the frontline

1. The problem I’m having:

I have Caddy behind Netlify using their proxy feature, and in front of some services in the local network.

One of them is Gitea, but looks like the X-Forwarded-* headers are not being forwarded correctly from the client, as when I open the login page there is a warning about it.

The proxy chain is

  1. gitea.kroltan.me (Netlify)
  2. home.kroltan.me:3003 (Caddy)
  3. riebeck.lan:3000 (Gitea)

2. Error messages and/or full log output:

The detected web site URL is “https://home.kroltan.me:3003/”, it’s unlikely matching the site config.
Mismatched app.ini ROOT_URL or reverse proxy “Host/X-Forwarded-Proto” config might cause wrong URL links for web UI/mail content/webhook notification/OAuth2 sign-in.

I understand this is the Caddy community I’m asking, I don’t want to make changes to Gitea to hide the warning. I want to adjust the server configuration to fix the underlying problem.

3. Caddy version:

v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0=

4. How I installed and ran Caddy:

FROM docker.io/caddy:2-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/netlify \
    --with github.com/ggicci/caddy-jwt

FROM docker.io/caddy:2-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

4a. System environment:

Up-to-date Fedora Server, Podman 5.8.2

4b. Command:

If you mean literally what ended up running, then:

caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

But I’m using Podman so it was really systemctl --user start caddy.service

4c. Service/unit/compose file:

[Unit]
Wants=network-online.target

[Container]
Image=caddy.build
Volume=%h/caddy/data:/data:Z
Volume=%h/caddy/conf:/etc/caddy:ro,Z
PublishPort=8080:8080
PublishPort=3003:3003
PublishPort=44333:44333
Network=caddy.network
ContainerName=caddy
Secret=CADDY_JWS_KEY,type=env,target=JWS_KEY

[Install]
WantedBy=default.target

4d. My complete Caddy config:

I have a couple utility includes and the main configuration files:

Caddyfile

# https://caddyserver.com/docs/caddyfile
{
	http_port 8080
	https_port 44333
}

http://* {
	redir https://{host}:44333 308
}

# As an alternative to editing the above site block, you can add your own site
# block files in the Caddyfile.d directory, and they will be included as long
# as they use the .caddyfile extension.
import Caddyfile.d/*.caddyfile

public_reverse_proxy

route {
	jwtauth {
		sign_key {$JWS_KEY}
		sign_alg HS256
		user_claims netlify_id
		from_header x-nf-sign
		issuer_whitelist netlify
	}

	reverse_proxy {args[0]} {
		# Only acceptable because we have authenticated Netlify above
		trusted_proxies 0.0.0.0/0 ::/0
	}
}

Caddyfile.d/forwards.caddyfile

https://home.kroltan.me:3003 {
	# Gitea
	import ../public_reverse_proxy http://riebeck.lan:3000
}

5. What I already tried, and links to relevant resources:

According to the reverse_proxy documentation, these headers are intentionally not propagated because they could be spoofed by a client, such as in the case the Caddy server is in fact public-facing.

Now, my server is only kind-of public facing, it has to be because I don’t have privileged port access on my ISP, so I have to put Netlify in front to get a nice URL. However, I don’t expect users to ever need to hit Caddy directly.

So I set up signed proxy redirects on Netlify, and added the jwtauth module to Caddy to ensure I am only serving requests coming through Netlify. That part is working! (I have verified by getting a 401 when trying to access the Caddy address directly from my browser)

However, even with the blanket trusted_proxiesassignment, it seems like the header is still not being properly forwarded. I’m not sure how to debug this either.

Assistance disclosure

No AI used.

Oh well, in the most massive rubber-duck-debugging moment of the year, I have immediately found the issue.

It was not the Caddy configuration that is incorrect, but on the Gitea side had its ROOT_URL set to http, not https.

I’ll leave the post around because this signed proxy deal is kinda neat and maybe one day someone is on the same stupid situation as me and finds it useful.