BASIC_AUTH_PASSWORD_HASHED=$(caddy hash-password --plaintext $BASIC_AUTH_PASSWORD) caddy run --config /etc/caddy/Caddyfile.json
3. The question I have:
Here are two excerpts from the latest Caddy release v2.5.0-rc.1:
Reverse proxy: The X-Forwarded-Host header will now be automatically set, along with X-Forwarded-For and X-Forwarded-Proto.
Reverse proxy: Incoming X-Forwarded-* headers will no longer be automatically trusted, to prevent spoofing. Now, trusted_proxies must be configured to specify a list of downstream proxies which are trusted to have sent good values. You only need to configure trusted proxies if Caddy is not the first server being connected to. For example, if you have Cloudflare in front of Caddy, then you should configure this with Cloudflare’s list of IP ranges.
I’m a little confused about them and they seem to be conflicting to me?
My questions are:
If I don’t set trusted_proxies will any X-Forwarded headers be passed to my Node.js application?
In which scenarios specifically are X-Forwarded-Host, X-Forwarded-For and X-Forwarded-Proto set?
4. Relevant information:
I run Caddy behind a bunch of proxies that do various things like caching and WAF.
Based on the answers above, this change may not be compatible with Section which is (a product simply described as) similar to CloudFlare, due to Section not being able to provide a list of IPs.
If I don’t set trusted_proxies will any X-Forwarded headers be passed to my Node.js application?
In which scenarios specifically are X-Forwarded-Host, X-Forwarded-For and X-Forwarded-Proto set?
If trusted_proxies is not set, Caddy will remove previous values and set X-Forwarded-Host, X-Forwarded-For, and X-Forwarded-Proto using the details from the time of the request. So in your case, it would use the Host header set by Cloudflare and the IP of CloudFlare’s edge host, which isn’t entirely useful.
In cases like this where you don’t have a list of IP ranges, the best way to work around this would likely be to authenticate CloudFlare via a different method, then set the trusted_proxies to 0.0.0.0/0. CloudFlare has a feature called Authenticated origin pulls, which has their edge present a client certificate to your origin. You can enable that and use a client_auth block in your Caddyfile to require verification. As long as client auth uses the require_and_verify mode, I would not hesitate to open up the range of trusted proxies.
This blog post seems to cover enabling authenticated origin pulls and configuring client_auth pretty well, though I haven’t tried that example specifically.
What @patrickeasters wrote is correct, but I’ll try to clarify the release notes further
This just means that, in prior versions, we only used to set X-Forwarded-For and X-Forwarded-Proto automatically, but now as of v2.5.0, we’re also setting X-Forwarded-Host by default. For most users, this shouldn’t change much, but it might save you a line of config if you were already setting X-Forwarded-Host yourself in your config.
In prior versions, when Caddy was behind another proxy (e.g. Cloudflare, etc), it would look at and use the values of the X-Forwarded-For and X-Forwarded-Proto headers that came on the request (assuming they exist on the request) assuming that the values were set by a trusted proxy.
The problem is that the request actually could have come from anywhere, i.e. directly from a client/attacker who could spoof those values by putting in whatever they wanted, then Caddy’s reverse_proxy would pass that along to your upstream app (in the case of X-Forwarded-For, with the connection’s remote address appended to that header, as proxies are meant to do). So an attacker could set the header to a different IP address than their own to walk around any kind of IP rate limiting logic you might have in your app, etc.
This obviously isn’t a safe default, so we’ve changed it to not trusting the values by default, so Caddy will ignore the incoming X-Forwarded-* headers from a downstream proxy (like Cloudflare etc). To get the previous behaviour, you must configure trusted_proxies with IP CIDRs of downstreams you trust to have correctly set those headers. You could just #yolo and set it to 0.0.0.0/0 as @patrickeasters said, but that turns off any header spoofing protection, if your server can be connected to directly.
And as @patrickeasters said, client authentication is another way to authenticate that the downstream proxy is trusted, if an IP allow-list isn’t appropriate for your usecase.
Point is, don’t trust direct requests from clients if you’re also putting another proxy in front of Caddy. If there’s no proxy in front of Caddy, then the default behaviour is safe, because Caddy will only use aspects of the incoming request/connection that couldn’t have been tampered with; i.e. the Host header which should be already validated by the host matcher in your routes, the remote address on the TCP connection which can’t be changed by the client else they never get a response, and the protocol which a factor of simply whether the connection was with TLS or not.
@patrickeasters and @francislavoie, thanks for the explanations. I’m not using CloudFlare so those links and information don’t specifically apply - although the concepts are similar to what I can use.
I’m already using basicauth between Section and Caddy to verify the request has passed through Section (Section appends the basic auth header before proxying the request through to my origin servers). So, I guess it is somewhat safe to use the basic auth setup and 0.0.0.0/0 to allow Caddy to pass those headers on.
I didn’t actually know 0.0.0.0/0 would work for this and would allow Caddy to pass those headers on. How does one come to know this information? Is this something that needs to be put in the documentation for this change?
Thanks for your help! I’ll have a shot at implementation now.
Well, the docs assume you understand how IP CIDRs work.
It’s not the job of the docs to explain how every bit of the internet works; that would increase the maintenance burden of the docs, and would make them overly verbose. The job of the docs is to explain how Caddy works, and its up to you to determine how to use that information according to your own needs (or come to the forums for help otherwise, basically).
Reverse proxy: Incoming X-Forwarded-* headers will no longer be automatically trusted, to prevent spoofing. Now, trusted_proxies must be configured to specify a list of downstream proxies which are trusted to have sent good values. You only need to configure trusted proxies if Caddy is not the first server being connected to. For example, if you have Cloudflare in front of Caddy, then you should configure this with Cloudflare’s list of IP ranges.