i may be wrong but i think what @francislavoie was saying is that it could be due to Caddyfile default ordering processes basicauth before route…
what you should do in this case is have the first lines of your Caddyfile to say something like:
{
order realip before basicauth
}
try this and follow up! i use caddy-auth-portal which obviously is processed after all internal Caddy configs so thats why i never ran into this issue.
edit:
just saw that you’ve done this already. sorry. in that case, @matt or @francislavoie may have to step back in, im drawing a blank
Yeah - I don’t know where the 401 is coming from so I couldn’t say.
Is the 401 from client_auth? If so I think it might be impossible to get the IP for those because that happens before Caddy executes its handlers, unless you used a listener_wrapper plugin instead (which AFAIK doesn’t exist).
The second you access secure-example.xnaas.info, the 401 appears in logs. If you put failed username/passwords in over and over, it’ll continue to be 401s with Cloudflare IPs. The real IP is only revealed upon successful login (HTTP 200).
So you see that above this section, if there’s no error, Caddy returns. But if there’s an error – which is the case with basicauth because it returns an actual caddyhttp.Error(http.StatusUnauthorized, ...) – it resets the RemoteAddr on the request before it gets logged.
@matt I think we may want to rethink how this is handled, because there is a legitimate usecase for wanting the RemoteAddr to be permanently modified here.
Or… now that I think of it, maybe the realip plugin should also mutate origReq := r.Context().Value(OriginalRequestCtxKey).(http.Request) so that Caddy doesn’t need to change? But that does feel like too funky of a fix (would introduce a weird hidden behaviour).
Edit: Oh aaaaaactually, what if you add realip to handle_errors?
common_log (which I think realip reads off of) uses the remote_addr field instead of X-Forwarded-For.
Edit: Actually, further log reading shows it to be wildly inconsistent…sometimes the common_log (or formatted) has the X-Forwarded-For address and sometimes it has remote_addr.
What realip does is actually modify r.RemoteAddr to have the value of X-Forwarded-For. But Caddy is resetting that change if an error occurred.
realip knows nothing of the logs at all. That’s not its job.
If you just used it as I wrote it, it probably wouldn’t work - you need to specify the same subdirectives to it as you did before, I just wrote that as a concept. I’d recommend pulling your realip into a snippet that you can plug into both your site block and your handle_errors block.
But yeah idk what to say past this point, but there’s clearly a problem with the order of operations here that causes errors being logged to have their RemoteAddr reset.
Just catching up… I will try to look into it soon. In the meantime, could you also reach out to the developer of realip and see if it’s intended to work in error routes too? My understanding is that it just sets RemoteAddr to the value of X-Forwarded-For. Since headers aren’t changed when invoking the error routes, this should still work in the handle_errors block.
Just to add searchable information here for others in the future, I ended up getting frustrated and dedicating the time to just parsing Caddy’s JSON logging.
NOTE: I use Authenticated Origin Pulls so my site cannot be accessed via my IP and only through Cloudflare. I also have a static IP.
[Definition]
# Option: actionstart
# Notes.: command executed once at the start of Fail2Ban.
# Values: CMD
#
actionstart =
# Option: actionstop
# Notes.: command executed once at the end of Fail2Ban
# Values: CMD
#
actionstop =
# Option: actioncheck
# Notes.: command executed once before each actionban command
# Values: CMD
#
actioncheck =
# Option: actionban
# Notes.: command executed when banning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: IP address
# number of failures
# unix timestamp of the ban time
# Values: CMD
actionban = curl -s -o /dev/null -X POST -H 'X-Auth-Email: <cfemail>' -H 'X-Auth-Key: <cftoken>' \
-H 'Content-Type: application/json' -d '{ "mode": "block", "configuration": { "target": "ip", "value": "<ip>" } }' \
https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules
# Option: actionunban
# Notes.: command executed when unbanning an IP. Take care that the
# command is executed with Fail2Ban user rights.
# Tags: IP address
# number of failures
# unix timestamp of the ban time
# Values: CMD
#
actionunban = curl -s -o /dev/null -X DELETE -H 'X-Auth-Email: <cfemail>' -H 'X-Auth-Key: <cftoken>' \
https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules/$(curl -s -X GET -H 'X-Auth-Email: <cfemail>' -H 'X-Auth-Key: <cftoken>' \
'https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules?mode=block&configuration_target=ip&configuration_value=<ip>&page=1&per_page=1' | tr -d '\n' | cut -d'"' -f6)
[Init]
cfemail = YOUR_CF_EMAIL
cftoken = YOUR_GLOBAL_TOKEN
I think there’s room for improvement somewhere, probably, but I’ve tested that this works just fine for now.
I’ve heard of jq before (and even tried to use it with huginn once upon a time), but I think that’s just even more added complexity to this than I want haha
Edit: This has already become incredibly non-portable as it is…
My module simply overwrites RemoteAddr before passing the request to caddyhttp.ServeHTTP
What is happening in this case is that basicauth requests are passed along as usual except once there, the overwritten RemoteAdddr is re-overwritten with the very original IP since its flagged with an error as shown here:
So without changing everything about how its working right now, the easiest solution might be for me to change BOTH request RemoteAddr values before passing to ServeHTTP. Something similar to:
origReq := r.Context().Value(OriginalRequestCtxKey).(http.Request)
req.RemoteAddr = net.JoinHostPort(parts[len(parts)-1], port)
origReq.RemoteAddr = net.JoinHostPort(parts[len(parts)-1], port)
for i := len(parts) - 1; i >= 0; i-- {
req.RemoteAddr = net.JoinHostPort(parts[i], port)
origReq.RemoteAddr = net.JoinHostPort(parts[i], port)
