Caddy - external is good but internal is failing

Help
1

1. The problem I’m having:

I have many web services setup. With Caddy everything is working as expected EXTERNALLY. But when I create static DNS records on my firewall so these sites will all load internally I’m not getting anything to load.
When inside the network:
ping juniper2.pokescans.ca PING juniper2.pokescans.ca (192.168.3.120) 56(84) bytes of data. 64 bytes from pokescans.ca (192.168.3.120): icmp_seq=1 ttl=64 time=0.165 ms 64 bytes from pokescans.ca (192.168.3.120): icmp_seq=2 ttl=64 time=0.163 ms

When I try to visit the site it cannot connect, output of curl shows it does connect to SOMETHING but it’s not the proper site/port (you can see in the caddy file the correct port is 9060).

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 192.168.3.120:80...
* Connected to juniper2.pokescans.ca (192.168.3.120) port 80 (#0)
> GET / HTTP/1.1
> Host: juniper2.pokescans.ca
> User-Agent: curl/7.88.1
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.26.3
< Date: Fri, 23 May 2025 21:07:07 GMT
< Content-Type: text/html
< Content-Length: 1415079
< Last-Modified: Sun, 29 Sep 2024 22:08:44 GMT
< Connection: keep-alive
< ETag: "66f9cfec-1597a7"
< Accept-Ranges: bytes
<

3. Caddy version:

os-caddy 2.0.0
opnsense 25.1.7_2

4. How I installed and ran Caddy

Caddy Plug-in in Opnsense

a. System environment:

Opnsense (FreeBSD)

d. My complete Caddy config:

# DO NOT EDIT THIS FILE -- OPNsense auto-generated file


# caddy_user=root

# Global Options
{
	log {
		output net unixgram//var/run/caddy/log.sock {
		}
		format json {
			time_format rfc3339
		}
		level DEBUG
	}

	servers {
		protocols h1 h2 h3
		log_credentials
	}

	dynamic_dns {
		provider cloudflare removed
		domains {
			stuartmedia.ca @
			ha.stuartmedia.ca @
			plex.stuartmedia.ca @
			storage.stuartmedia.ca @
		}
		versions ipv4
	}

	email trev@stuartmedia.ca
	grace_period 10s
	import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


stuartmedia.ca {
	log {
		output file /var/log/caddy/access/9863b975-8fae-4a58-8c83-d1a51387239a.log {
			roll_keep_for 10d
		}
	}
	tls {
		issuer acme {
			dns cloudflare removed
		}
	}

	handle {
		reverse_proxy 192.168.3.101:5055 {
		}
	}
}

juniper2.pokescans.ca {
	log {
		output file /var/log/caddy/access/3aaa271f-8e8e-4a31-8286-7151626988dc.log {
			roll_keep_for 10d
		}
	}
	tls {
		issuer acme {
			dns cloudflare removed
		}
	}

	handle {
		reverse_proxy 192.168.3.120:9060 {
		}
	}
}

map2.pokescans.ca {
	log {
		output file /var/log/caddy/access/148f673c-8479-40c2-9f27-6c362493c636.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.120:8080 {
		}
	}
}

dragonite2.pokescans.ca {
	log {
		output file /var/log/caddy/access/407b792c-af3f-4280-acea-b3a51ef9dd9b.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.120:7273 {
		}
	}
}

koji2.pokescans.ca {
	log {
		output file /var/log/caddy/access/e1d74b7b-ff9f-4176-9195-e91853467a2e.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.120:8082 {
		}
	}
}

http://data.pokescans.ca {
	log {
		output file /var/log/caddy/access/d4b11b6f-f25a-4949-9504-8d3cd3a1051b.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.120:7070 {
		}
	}
}

rotom2.pokescans.ca {
	log {
		output file /var/log/caddy/access/e7456b08-e7d6-4b54-b0a0-1b5f30ca8038.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.120:7072 {
		}
	}
}

ha.stuartmedia.ca {
	log {
		output file /var/log/caddy/access/a3d94daa-9ea1-4c13-afde-ff3b8ccb77ae.log {
			roll_keep_for 10d
		}
	}
	tls {
		issuer acme {
			dns cloudflare removed
		}
	}

	handle {
		reverse_proxy 192.168.3.105:8123 {
		}
	}
}

plex.stuartmedia.ca {
	log {
		output file /var/log/caddy/access/cd139ccd-5488-4412-862a-496cd42e4b71.log {
			roll_keep_for 10d
		}
	}
	tls {
		issuer acme {
			dns cloudflare removed
		}
	}

	handle {
		reverse_proxy 192.168.3.101:32400 {
		}
	}
}

storage.stuartmedia.ca {
	log {
		output file /var/log/caddy/access/55f1d8ca-6024-4432-a32d-25d20c56ec73.log {
			roll_keep_for 10d
		}
	}
	tls {
		issuer acme {
			dns cloudflare removed
		}
	}

	handle {
		reverse_proxy 192.168.3.101:2283 {
			header_up Host {upstream_hostport}

			transport http {
			}
		}
	}
}

cent-poke.pokescans.ca {
	log {
		output file /var/log/caddy/access/5757bce5-7344-4863-9a8b-738781042c73.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.120:443 {
		}
	}
}

tiles2.pokescans.ca {
	log {
		output file /var/log/caddy/access/eab6a7c7-2535-4463-a024-5dfd58a30d55.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.120:9002 {
		}
	}
}

firiona.rgit.ca {
	log {
		output file /var/log/caddy/access/d680a629-aed8-45c3-ab80-10f5961be3a2.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.121:7634 {
		}
	}
}

http://passlist.org {
	log {
		output file /var/log/caddy/access/574d31c8-c296-41c1-839d-e6c34739050c.log {
			roll_keep_for 10d
		}
	}

	handle {
		reverse_proxy 192.168.3.121:7636 {
		}
	}
}

import /usr/local/etc/caddy/caddy.d/*.conf
2

That something is your NGINX. It says it right here:

You also mentioned that the correct port is 9060:

but it looks like you forgot to attach your Caddyfile.

It might also help if you describe how Caddy is exposed to the internet. Are you using port forwarding on your router and forwarding port 80 to Caddy, which is listening on port 9060? If that’s the case, then:

curl http://juniper2.pokescans.ca

would only work externally. For internal testing, you’d probably need:

curl http://juniper2.pokescans.ca:9060
3

OPNsense supports reflection and hairpin NAT. If I were in your shoes, I’d take advantage of that instead of overwriting DNS within your network.

You can find more details here:

An excerpt from the docs:

Introduction to Reflection and Hairpin NAT

For example, you have a Webserver example.com with the internal IP 172.16.1.1 in your DMZ. It has a public DNS Record of example.com in A 203.0.113.1.

Your internal client 192.168.1.1 can’t reach the Webserver if it resolves the DNS A-Record 203.0.113.1. When the OPNsense receives the packet from the client 192.168.1.1 with the destination IP 203.0.113.1, it chooses itself as the target, and not 172.16.1.1. That’s because the external IPv4 address 203.0.113.1 is mapped to the WAN interface of the OPNsense.

That’s where Reflection NAT comes into play. It creates NAT rules which help your internal client 192.168.1.1 to communicate with your webserver 203.0.113.1, by using the OPNsense as the “translator” to the actual destination 172.16.1.1.

In your case:

  • Their example.com = your juniper2.pokescans.ca
  • Their 203.0.113.1 = your 161.184.60.50
  • Their 172.16.1.1 = your 192.168.3.120 (just a guess on that last one)
4

Sorry about that, I was on a small laptop screen and didn’t actually look close enough at the preview, caddy file is there now.
I’ll explore the reflection and hairpin Nat you’ve suggested.

Also I thought with Caddy I wouldn’t need to provide the ports, internal or external. The websites in question DO load perfectly if I use http://192.168.3.120:9060

9

No worries :slight_smile: it happens! This clears things up:

juniper2.pokescans.ca {
	# ...

	handle {
		reverse_proxy 192.168.3.120:9060 {
		}
	}
}

Now I see that it’s not Caddy listening on 192.168.3.120:9060 - that’s actually the upstream for juniper2.pokescans.ca.

You’ve got NGINX listening on port 80 on your internal IP, so Caddy can’t use that port too. Only one service can bind to a port at a time.

Take a look at your OPNsense setup: if NGINX is already bound to port 80 internally, Caddy won’t be able to bind to it. You might want to try disabling NGINX and configuring Caddy to listen on both internal and external interfaces. Also, enabling debug in Caddy could help - it might show that there’s another service blocking port 80 on the internal IP.

At this point, it’s really more of an OPNsense issue than a Caddy one.

10

Nginx is running on the upstream server on port 80. Not on the opnsense server. At least I didn’t think it was on opnsense anywhere.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.