OPNSense + Wireguard + Caddy: connection closed

Help
1

1. The problem I’m having:

Hello, I use OPNSense with the Wireguard plugin at home (192.168.1.0/24). I connect to the home network with my cell phone via Wireguard (WG-opnsense: 10.10.0.1 WG-cellphone 10.10.0.2).

This works perfectly, even with local DNS via adguard routing. For example to immich.example.de (selfhosted).

Now I want to install and use the OPNSense Caddy plugin. It is initially a very simple configuration, exactly according to the instructions
https://docs.opnsense.org/manual/how-tos/caddy.html#caddy-reverse-proxy

This also works so far.
However, I would like to block access from the Internet. This also works according to the instructions Caddy: Reverse Proxy — OPNsense documentation . At home in the home network, I can access immich.example.de via caddy. This is not possible from the Internet.

But if I access it from my cell phone via Wireguard, it doesn’t work either. But it should work.

tl;dr
OPNSense and Wireguard working fine, but wg-clients can’t open caddy served website.

2. Error messages and/or full log output:

Cellphone Broswer: ERR_CONNECTION_CLOSED

3. Caddy version:

Actual OPNSense 25.1.a_194 with CaddyPlugin 1.6.3_1

4. How I installed and ran Caddy:

OPNSense in a own router (APU-Router), Caddy via Plugin

d. My complete Caddy config:

# caddy_user=root

# Global Options
{
	log {
		output net unixgram//var/run/caddy/log.sock {
		}
		format json {
			time_format rfc3339
		}
	}

	email example@example.de
	grace_period 10s
	import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Reverse Proxy Domain: "38ba0804-7c0a-4dc8-9b95-2c4acaXXXXX"
immich.example.de {
	@9acb4085-666d-4a2c-b32b-b6047faXXXX {
		client_ip 192.168.1.0/24 10.10.0.0/24 10.7.0.0/24 10.9.0.0/24 192.168.200.0/24 192.168.90.0/24 10.10.0.2/32
	}

	handle @9acb4085-666d-4a2c-b32b-b6047fXXXXX {
		handle {
			reverse_proxy 192.168.1.39:2283 {
			}
		}
	}

	abort
}

import /usr/local/etc/caddy/caddy.d/*.conf

5. Links to relevant resources:

edit: I’m solving it half:
I#; using the opnsense plugin AdGuard. If I redirect in adguard immich.example-de to its own opnsense IP (192.168.1.1), than it works?

2

Howdy @solear, welcome to the Caddy community.

Is bind (Caddyfile directive) — Caddy Documentation not suitable for this setup? You could configure Caddy to bind to the LAN and WireGuard interfaces rather than trying to match the client IP.

If that’s not doable, considering your configuration (check client IP → reverse proxy allowed clients upstream), we should probably double check something very basic to start with. What client_ip does Caddy actually think those requests are coming from?

For that…

We need the actual logs from Caddy, rather than the error from your mobile device.

The help template you filled out should’ve directed you to the debug option:

Please ENABLE DEBUG MODE FIRST by adding “debug” to the global options of your Caddyfile. See https://caddyserver.com/docs/caddyfile/options#debug for an example.

Ensuring you’re collecting those logs, making a request, and then posting the resultant logs here should give us a better idea of what might be going on.

1 Like
3

Caddy listens on all IP addresses on the OPNsense (any interface).

This means it will also listen on the IP address of any wireguard interface and any local interface. If the allowed IPs in Wireguard allow access to any of these IP addresses, and the Wireguard Firewall rules allow the connection, then it will establish to Caddy.

If the connection closes, its most likely a Wireguard MTU issue. Try to lower the MTU on the Client that connects by lowering the MTU in the Wireguard configuration file of the client.

Otherwise, yeah debug logs would help.

2 Likes
4

Hello! Thank you for your help and sorry for the late reply.

The caddy-debug-log is clear (empty).

Maybe this is expected behavior.

Again:
Opnsense with caddy plugin, immich.example.com works inside my network as expected (blocked outside my network).
In my home network, I can browse to immich.example.de or 192.168.1.50.
From my smartphone (connected via wireguard to my home network) immich.example.de doesnt work, but 192.168.1.50.

I also have the opnsense adguard-plugin. If I state there “immich.example.de => 192.168.1.1” than it does work over wireguard.

It seems that the routes for my domain are:
Inside my Home Network:
Browser => opnsense/caddy

Over wireguard: smartphone => wg =>opnsense/adguard => opnsense/caddy.

I don’t understand the logic.
In my mind wireguard ask at first adguard, and not caddy.
On the other hand in my home network, the browser hits at first opnsense/caddy.

I dont understand the logic in opnsense with two plugins adguard and caddy.

5

Whats the allowed IPs in wireguard?

(client and server, both sides please)

Essentially what should happen if its 0.0.0.0 on client:

Client resolves example.com to external IP of OPNsense WAN interface

Paket gets sent through wireguard tunnel

Firewall rule on Wireguard Group allows access to 443 on This Firewall.

There shouldnt be any Split DNS tricks needed.

Plesse read this: Caddy: Reverse Proxy — OPNsense documentation

1 Like
6

The allowed IPs in Wireguard:

OPNSense: 10.10.0.2/32 (=wg smartphone IP)
Smartphone: 10.10.0.0/24 192.168.1.0/24 (wg network, home network; Opnsense IP: 192.168.1.1)

Thank you for the opnsense/caddy manual. I‘ve already read this and I dont understand why it doesnt work like expected.

7

Check with a DNS app on the smartphone how example.com resolves when:

  • Wireguard is enabled
  • Wireguard is disabled

Since allowed IPs is not 0.0.0.0/0, any traffic that is not on the allowed IPs (10.10.0.0/24 192.168.1.0/24) will be routed to the default gateway of your phone (aka the internet).

This should essentially work if example.com resolves to the external IP of your OPNsense while connected to wireguard. If it does not, now you know where to look.

2 Likes
8

Thank you!
Its getting hotter :upside_down_face:
I dont wanna route all my traffic over my homenetwork (0.0.0.0/0 allowed IP).
But…I see 10.10.0.1 is my DNS. This way Im using my adguard via wg.

And the missing link is on opnsense between my wg server service 10.10.0.1 and caddy.
Thats why it works out of the box with adguard but not with caddy.
WG server dns is pointed to 192.168.1.1, but somehow its routes to adguard instead of caddy.

Its an bug in the wg plugin?

I understand now the background. And I can live with the dns rewrite in adguard to caddy.

Thank you very much! @Monviech I really appreache it.

9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.