Help with fail2ban behind double Caddy proxy

Help
1

1. Caddy version (caddy version):

2.4.3

2. How I run Caddy:

a. System environment:

Ubuntu, installed via Ansible (GitHub - caddy-ansible/caddy-ansible: Ansible role for installing and configuring the Caddy web server), running via systemd

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

;
; Ansible managed
;
; source: https://github.com/mholt/caddy/blob/master/dist/init/linux-systemd/caddy.service
; version: 6be0386
; changes: Set variables via Ansible

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target

[Service]
Restart=on-failure
StartLimitInterval=86400
StartLimitBurst=5

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy

ExecStart="/usr/local/bin/caddy" run --environ --config "/home/fuzzy/caddy/Caddyfile"
ExecReload="/usr/local/bin/caddy" reload --config "/home/fuzzy/caddy/Caddyfile"

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=false
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
; … except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy /var/log/caddy

; The following additional security directives only work with systemd v229 or later.
; They further retrict privileges that can be gained by caddy.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

Caddyfile #1

{
	log {
		output file /home/{{main_username}}/caddy/access.log {
			roll_size 100mb
			roll_keep 5
			roll_keep_for 168h
		}
	}
	email {{secret_email}}
}

(proxy_options) {
	header X-Real-IP {remote_host}
	header X-Forwarded-Proto {scheme}
}
(personal_headers) {
	header {
		Strict-Transport-Security "max-age=31536000; includeSubdomains"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		Referrer-Policy "same-origin"
		Content-Security-Policy "frame-ancestors {{secret_personal_url}} *.{{secret_personal_url}}"
		-Server
		Permissions-Policy "geolocation=(self {{secret_personal_url}} *.{{secret_personal_url}}), microphone=()"
	}
}
(no_robots) {
		respond /robots.txt 200 {
		body "User-agent: *
Disallow: /"
	}
}

*.{{secret_personal_url}} {
	tls /home/{{main_username}}/lego/certificates/_.{{secret_personal_url}}.crt /home/{{main_username}}/lego/certificates/_.{{secret_personal_url}}.key {
	}

	@smarthome host smarthome.{{secret_personal_url}}
	handle @smarthome {
		reverse_proxy 10.10.10.10:5315
		import proxy_options
		import personal_headers
		import no_robots
	}

	@dashboard host dashboard.{{secret_personal_url}}
	handle @dashboard {
		reverse_proxy 10.10.10.10:5100
		import proxy_options
		import personal_headers
		import no_robots
	}

	@photos host photos.{{secret_personal_url}}
	handle @photos {
		reverse_proxy 10.10.10.10:8000
		import proxy_options
		import personal_headers
		import no_robots
	}

	@wallabag host wallabag.{{secret_personal_url}}
	handle @wallabag {
		reverse_proxy 10.10.10.10:300
		import proxy_options
		import personal_headers
		import no_robots
	}

	# Fallback for otherwise unhandled domains
	handle {
		redir {{secret_redirect}}
	}
}

Caddyfile #2

{
	log {
		output file /home/{{main_username}}/caddy/access.log {
			roll_size 100mb
			roll_keep 5
			roll_keep_for 168h
		}
	}
	email {{secret_email}}
	auto_https off
}

(proxy_options) {
	header X-Real-IP {remote_host}
	header X-Forwarded-Proto {scheme}
}

(headers) {
	header {
		Strict-Transport-Security "max-age=31536000; includeSubdomains"
		X-XSS-Protection "1; mode=block"
		X-Content-Type-Options "nosniff"
		Referrer-Policy "same-origin"
		Content-Security-Policy "frame-ancestors {{secret_personal_url}} *.{{secret_personal_url}}"
		-Server
		Permissions-Policy "geolocation=(self {{secret_personal_url}} *.{{secret_personal_url}}), microphone=()"
	}
}

:5315 {
		reverse_proxy http://192.168.30.12:5315
		import headers
		import proxy_options
}
:5100 {
		reverse_proxy http://192.168.30.13:5100
		import headers
		import proxy_options
}
:8000 {
		reverse_proxy http://192.168.30.11:8000
		import headers
		import proxy_options
}
:300 {
		reverse_proxy http://192.168.30.13:300
		import headers
		import proxy_options
}

3. The problem I’m having:

Bit of a long leadup but bear with me. I’m trying to create a set up where all my traffic is routed through a VPS and then through a Wireguard tunnel to my homelab. Goal is to avoid punching holes in my home firewall (including 80/443). The VPS runs Caddy and is the first Caddyfile above. I then would have a server running a second Caddy instance sitting in a DMZ to route the traffic to the appropriate server. I run a bunch of things in separated LXC/VMs. Here’s an image that shows what I’m doing:

Reverse Proxy Setup
Reverse Proxy Setup795Ă—410 46.1 KB

The issue is I also want to run fail2ban on the app servers to ban any bots/malicious actors. I may use a cloud firewall from my VPS provider to make sure the firewall blocks the traffic at the source (I’ll figure that part out separately). My issue is that right now is that in the logs for some apps the IP address is either the IP address of the Caddy Server #2 (ie 192.168.10.5) or is the Wireguard ip address of Caddy Server #1 (10.10.10.10). Not sure if it’s an issue with the app or what, because when I look at the Caddy log the X-Forward-For IP address is the IP address i’d want to ban (ie in this case my home IP as that’s what I’m testing from).

Perhaps the solution here is to rely on the Caddy log for the IP address to ban? I have a link to a thread down below that mentions looking into that from March but I don’t see any follow up and the thread is locked. Or is there a header/something I’ve misconfigured above that would lead to the “wrong” IP address showing up in the app logs? Would the “real ip” plugin help here?

4. Error messages and/or full log output:

Not really relevant I don’t think.

5. What I already tried:

You can see in my Caddyfile I’ve tried to pass through any relevant headers with IP addresses, but sometimes it’s still the wrong IP address seen.

6. Links to relevant resources:

2

The CADDYPATH environment variable doesn’t apply for Caddy v2, by the way. That’s a Caddy v1 variable, if I remember correctly.

Caddy v2 will save it in the $HOME of the user it runs as (i.e. www-data), whatever that is on your system. See the docs on the paths Caddy uses:

I figure you should report this to the caddy-ansible repo, since you took it from there. That’s an unofficial installation method.

This isn’t correct, this will set response headers. Those are request headers that are typically sent to proxy upstreams.

Either way, you shouldn’t need this, because Caddy will set the X-Forwarded-For and X-Forwarded-Proto headers for you automatically:

You should make sure your upstream apps are set up to read from X-Forwarded-For. Some apps require some configuration to “trust” the requests from the incoming proxy (i.e. if the “remote address” of the request is in some whitelist, then allow reading from X-Forwarded-For for the real client IP).

If you’re relying on Caddy’s own logs for doing IP banning, then yeah, the realip plugin might help. Basically what that plugin does, is it takes the value from X-Forwarded-For and stuffs it on the http.Request object inside Caddy to overwrite the RemoteAddr which is typically the IP address attached to the TCP packet.

As an aside, I think what would be “easier” is instead of using random port numbers for each service, use a site address like http://subdomain.example.com on your 2nd Caddy, and proxy over port 80 instead (if possible). That way you retain the name of the service.

Routing based on hostname still works over HTTP, because Caddy will set the Host header on the proxy request to the same one it received from the client.

You could probably just do reverse_proxy 10.10.10.10:80 on your first Caddy, don’t do any host matching, then do the actual host matching on the upstream app. (You could still whitelist the subdomains and serve a redirect in your first instance if you like).

1 Like
3

Thank you, I’ll open an issue over there and link to here.
EDIT: Actually looks like it was discussed here: CADDYPATH removed in v2 · Issue #19 · caddy-ansible/caddy-ansible · GitHub

Ah, I think I’d read that link at one point but missed that it covered those as well. I’ll take it out.

I’ll look into this.

Ahh, that makes sense. I couldn’t quite tell what it was for, seemed more geared towards if I was using something like Cloudflare which I know “obfuscates” the real IP address and I had to do some “magic” when I was using NGINX/Cloudflare in the past.

I’ve been looking into doing local DNS stuff, just haven’t gotten there. I do have comments in my Caddyfile to remind me what the port is. So what you’re saying is I could just have something like this in the first Caddyfile:

	@wallabag host wallabag.example.com
	handle @wallabag {
		reverse_proxy 10.10.10.10:80
		import proxy_options
		import personal_headers
		import no_robots
	}

and then this in the second?

wallabag.example.com {
		reverse_proxy http://192.168.30.13:300
		import headers
}

That would be nice.

4

You actually don’t need to worry about local DNS for this. This is just about Caddy doing string comparisons on the Host header, actual DNS queries are not involved for host matching.

In the second, it would have to be like this:

http://wallabag.example.com {
	reverse_proxy http://192.168.30.13:300
	import headers
}

The http:// in front matters, because it tells Caddy to listen on port 80 for HTTP requests. Otherwise, Caddy would try to enable Automatic HTTPS for that domain (which would fail because ACME challenges requests would not reach, since your first Caddy would intercept them) and listen on port 443.

1 Like
5

Ok thank you for the help on that, I redid my Caddyfile and it works, though oddly reloads now take a long time. Not sure if it’s related, will keep digging.

Still having issues with X-Fowarded-For. As a test, I spun up “whoami” container on Caddy Server #2 with this in my Caddyfile:

:49152 {
        reverse_proxy localhost:49153
}

I put this in my Caddy Server #1 Caddyfile:

:49152 {
        reverse_proxy 10.10.10.10:49152
}

I then ran curl "http://10.10.10.10:49152" and got this in response:

Hostname: d0a46a01201c
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 172.17.0.1:60934
GET / HTTP/1.1
Host: 10.10.10.10:49152
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 10.10.10.1
X-Forwarded-Proto: http

That’s what I see in the logs in Jellyfin, Jellyfin.Server.Implementations.Users.UserManager: Authentication request for "test" has been denied (IP: "10.10.10.1"). Is there a way for me to modify the X-Forwarded-For header in any way to strip the first IP address of my VPN off?

6

You can turn on debug mode in Caddy to see the headers being passed through it (global option).

I’m not totally sure which IP is which at this point, I’m not sure which IP you expect to have at the other end.

But the way Caddy’s reverse_proxy works, it’ll append the current client’s RemoteAddr to an existing X-Forwarded-For header if one came in on the request. A properly written app should be able to handle that without issue (it should look at the first IP in the header to get the real client IP).

But if the app has issues with that, you can probably override that behaviour with header_up like this, I think:

reverse_proxy localhost:49153 {
	header_up X-Forwarded-For {header.X-Forwarded-For}
}

Although I think that might not work because of the order of operations in Caddy (I think the reverse_proxy module manipulates the X-Forwarded-For header on the request before header_up runs, so this might be a no-op)… if that’s the case I’m not sure what to suggest. It might involve the request_header to copy the original value to some other header temporarily, overwriting X-Forwarded-For with that temporary header (as above), then removing the temporary header with header_up -Temporary or something like that. Kinda wacky. I hope that’s not necessary :joy:

1 Like
7

Sorry if I haven’t been clear. My VPN’s IP is 10.10.10.1 and let’s just say my actual IP is 123.123.123.123. I’d obviously want to ban 123.123.123.123 and not 10.10.10.1 because banning the VPN address would ban everyone/everything from being able to access all my services, not just the single “bad” IP address.

Yeah I see the IP address I want to ban in one or two apps, but sadly not in something like Jellyfin which surprises me a bit. And I’d think the “whoami” container would show all the IPs, not just the VPN’s.

Didn’t work :frowning:

I’ll keep playing around on my end/check out debugging mode. Didn’t have much time this morning for more than a quick check of the header_up option.

8

I did some more debugging just now setting up NGINX and Caddy to do the same basic curl to the whoami container:

NGINX:

Hostname: d0a46a01201c
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 172.17.0.1:48460
GET / HTTP/1.1
Host: test.example.com
User-Agent: curl/7.68.0
Accept: */*
Connection: close
X-Forwarded-For: 123.123.123.123
X-Forwarded-Proto: http
X-Real-Ip: 123.123.123.123

Caddy:

Hostname: d0a46a01201c
IP: 127.0.0.1
IP: 172.17.0.2
RemoteAddr: 172.17.0.1:42974
GET / HTTP/1.1
Host: test.example.com
User-Agent: curl/7.68.0
Accept: */*
Accept-Encoding: gzip
X-Forwarded-For: 123.123.123.123, 10.10.10.1
X-Forwarded-Proto: http

So it looks like I need the realip plugin to get what I want/need. Will give that a try later.

9

Doing this would give you the same result as nginx gave you above:

request_header X-Real-Ip {header.X-Forwarded-For}
reverse_proxy localhost:49153 {
	header_up X-Forwarded-For {header.X-Real-Ip}
}

Basically copies the X-Forwarded-For from the original request into X-Real-Ip, then starts the proxy, and overwrites X-Forwarded-For with the value of X-Real-Ip from earlier (essentially what I explained earlier)

1 Like
10

Turns out you were right and it was mostly an app configuration issue. Had to set trusted proxies and all is well. Thank you so much for all your help, really appreciate it!

2 Likes
11

This topic was automatically closed after 30 days. New replies are no longer allowed.