Reverse proxy works in homelab but not in external server

Help
6

@timelordx

I updated my Caddyfile to:

  • use the letsencrypt staging environment: Staging Environment - Let's Encrypt
  • simplify the redirection
  • stick with firewall listening to low ports and redirecting to caddy on high ports

pf.conf (firewall)

rdr pass on $ext_if inet proto tcp from any to ($ext_if) port 80 -> $load_balancer port 8080
rdr pass on $ext_if inet proto tcp from any to ($ext_if) port 443 -> $load_balancer port 8443

Caddyfile

{
        http_port 8080
        https_port 8443
        acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
...
dev.apps.generalwildfireservices.com {
	import logging

	route /tiles* {
		reverse_proxy 10.0.0.40:3000
	}
	route /app1* {
		reverse_proxy 10.0.0.30:8080
	}
}

Everything is working fine now. I think the root error was that I blew through my rate limits on letsencrypt while I was developing this load balancer. This happened b/c I was developing this in a FreeBSD jail, and every time I cycled the jail, it had to re-request the acme creds. I could see in the logs that letsencrypt was refusing me any further creds for some time period.

So the useful fixes are:

  • mount external state (directory) into the load balancer jail for storing acme creds. In FreeBSD this means mounting a directory from the HOST as a nullfs mount within the jail.
  • use the lets encrypt staging url until my account resets it’s rate limits
  • simplify the Caddyfile. b/c I specify the http(s)_port (s) as 8080/8443, then I can just specify the site, without the port, in the site block / site address field.

Thanks again @timelordx!