Caddy on OPNSense with subnet, partial website loads

Help
1

1. The problem I’m having:

EDIT - See latest comment. The services running in Docker on Proxmox are fine. But running outside of Docker any request for large files via Caddy hangs.

I am running Caddy on OPNSense with the os-caddy plugin. I am trying to use it to reverse proxy services running on a Proxmox server, which is on a separate machine and a separate vlan/subnet.

I previously had Caddy running directly on Proxmox for all of these services without issue. If I switch from os-caddy back to the Caddy on Proxmox everything works fine. The caddy configs are similar. I can still connect to all of these services both from the client and from the OPNsense server.

I am able to use curl -vL successfully for all of the websites (from multiple clients/opnsense/local vlans), but in the browser most only partially load and eventually time out. I have tried several clients/browsers and tried to rule out any caching issues (including by moving to new domains). I believe there are no other connection issues between the machines.

Unfortunately, I have not figured out any common patterns for the ones that load vs ones that time out (most time out). The ones that work and don’t work are consistent and they fail to load in a consistent way.

I also had os-caddy proxying three services not on Proxmox: OPNsense WebUI, AdGuard Home on OPNsense, Managed Switch Web UI. These are all within the same subnet as the router, and none have issues (these are not reflected in config below because I wanted as simple case as possible). This led me to believe its something about Proxmox being on a separate subnet/vlan but have only reached dead ends.

I have done my best to rule out any network issues. And the lack of connection issues and also the fact that I can use caddy on proxmox directly without issue makes me believe its not a network issue. But I haven’t tested putting proxmox back on the same subnet as router or similar.

Network is roughly:

____________
| OPNSense  |  
| os-caddy  |  -- [managed switch] -- [proxmox] (10.0.10.x)
(192.168.1.1)            |
                         |
                     [clients] (192.168.x.x)

2. Error messages and/or full log output:

Homepage is one of the services that partially loads. This is as far as it gets, everything else times out:

Screenshot from 2024-09-06 19-25-39
Screenshot from 2024-09-06 19-25-391915×1013 154 KB

This is an example of the error log I get when a request times out:

{
  "ts": "2024-09-07T01:53:47Z",
  "logger": "http.log.access.f76fb375-99c1-465b-977b-4de404498305",
  "msg": "handled request",
  "request": {
    "remote_ip": "192.168.1.100",
    "remote_port": "46540",
    "client_ip": "192.168.1.100",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "maple.example.com",
    "uri": "/_app/immutable/chunks/handle-error.Vcm-X60N.js",
    "headers": {
      "Accept-Language": [
        "en-US,en;q=0.5"
      ],
      "Sec-Fetch-Dest": [
        "script"
      ],
      "Sec-Fetch-Mode": [
        "cors"
      ],
      "Sec-Fetch-Site": [
        "same-origin"
      ],
      "Priority": [
        "u=1"
      ],
      "Accept": [
        "*/*"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br, zstd"
      ],
      "Dnt": [
        "1"
      ],
      "Sec-Gpc": [
        "1"
      ],
      "Referer": [
        "https://maple.example.com/"
      ],
      "Te": [
        "trailers"
      ],
      "User-Agent": [
        "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "maple.example.com"
    }
  },
  "bytes_read": 0,
  "user_id": "",
  "duration": 0.010085252,
  "size": 472,
  "status": 200,
  "resp_headers": {
    "X-Powered-By": [
      "Express"
    ],
    "Last-Modified": [
      "Wed, 10 Jul 2024 16:31:12 GMT"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ],
    "Content-Type": [
      "text/javascript"
    ],
    "Content-Encoding": [
      "br"
    ],
    "Date": [
      "Sat, 07 Sep 2024 01:53:47 GMT"
    ],
    "Cache-Control": [
      "public,max-age=31536000,immutable"
    ],
    "Server": [
      "Caddy"
    ],
    "Etag": [
      "W/\"472-1720629072000\""
    ],
    "Content-Length": [
      "472"
    ],
    "Vary": [
      "Accept-Encoding"
    ]
  }
}

And another error:

{
  "ts": "2024-09-07T02:07:40Z",
  "logger": "http.log.access.d2de8d66-ef3d-43ac-a199-f2b872129c2a",
  "msg": "handled request",
  "request": {
    "remote_ip": "192.168.1.100",
    "remote_port": "34288",
    "client_ip": "192.168.1.100",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "maple2.example.com",
    "uri": "/_next/static/media/Manrope.63012343.woff2",
    "headers": {
      "User-Agent": [
        "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"
      ],
      "Dnt": [
        "1"
      ],
      "Referer": [
        "https://maple2.example.com/_next/static/css/dd916459193df04c.css"
      ],
      "Sec-Fetch-Dest": [
        "font"
      ],
      "Sec-Fetch-Site": [
        "same-origin"
      ],
      "Accept": [
        "application/font-woff2;q=1.0,application/font-woff;q=0.9,*/*;q=0.8"
      ],
      "Accept-Language": [
        "en-US,en;q=0.5"
      ],
      "Accept-Encoding": [
        "identity"
      ],
      "Sec-Gpc": [
        "1"
      ],
      "Sec-Fetch-Mode": [
        "cors"
      ],
      "Te": [
        "trailers"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "maple2.example.com"
    }
  },
  "bytes_read": 0,
  "user_id": "",
  "duration": 250.197089714,
  "size": 0,
  "status": 502,
  "resp_headers": {
    "Server": [
      "Caddy"
    ],
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ]
  }
}

Immich is one of the services that works without issue. I am able to load it, view and download photos. Example log of successful photo download from Immich via os-caddy proxy.

{
  "ts": "2024-09-07T02:06:04Z",
  "logger": "http.log.access.f76fb375-99c1-465b-977b-4de404498305",
  "msg": "handled request",
  "request": {
    "remote_ip": "192.168.1.100",
    "remote_port": "33556",
    "client_ip": "192.168.1.100",
    "proto": "HTTP/2.0",
    "method": "GET",
    "host": "maple.example.com",
    "uri": "/api/assets/91e6c808-3b57-459b-b43b-d7552adab14b/original",
    "headers": {
      "Sec-Fetch-Mode": [
        "cors"
      ],
      "User-Agent": [
        "Mozilla/5.0 (X11; Linux x86_64; rv:130.0) Gecko/20100101 Firefox/130.0"
      ],
      "Sec-Gpc": [
        "1"
      ],
      "Sec-Fetch-Site": [
        "same-origin"
      ],
      "Priority": [
        "u=0"
      ],
      "Te": [
        "trailers"
      ],
      "Referer": [
        "https://maple.example.com/photos/91e6c808-3b57-459b-b43b-d7552adab14b"
      ],
      "Sec-Fetch-Dest": [
        "empty"
      ],
      "Accept": [
        "*/*"
      ],
      "Dnt": [
        "1"
      ],
      "Cookie": [
        "REDACTED"
      ],
      "Accept-Language": [
        "en-US,en;q=0.5"
      ],
      "Accept-Encoding": [
        "gzip, deflate, br, zstd"
      ]
    },
    "tls": {
      "resumed": false,
      "version": 772,
      "cipher_suite": 4865,
      "proto": "h2",
      "server_name": "maple.example.com"
    }
  },
  "bytes_read": 0,
  "user_id": "",
  "duration": 0.865212044,
  "size": 4392801,
  "status": 200,
  "resp_headers": {
    "Alt-Svc": [
      "h3=\":443\"; ma=2592000"
    ],
    "Date": [
      "Sat, 07 Sep 2024 02:06:03 GMT"
    ],
    "Cache-Control": [
      "private, max-age=86400, no-transform"
    ],
    "X-Powered-By": [
      "Express"
    ],
    "Accept-Ranges": [
      "bytes"
    ],
    "Server": [
      "Caddy"
    ],
    "Etag": [
      "W/\"430761-191c3884f40\""
    ],
    "Last-Modified": [
      "Thu, 05 Sep 2024 18:52:56 GMT"
    ],
    "Content-Length": [
      "4392801"
    ],
    "Content-Type": [
      "image/jpeg"
    ],
    "X-Immich-Cid": [
      "x84jjii5"
    ]
  }
}

3. Caddy version:

The OPNSense packages installed:

caddy-custom 2.8.4.3.0.4.2.3_3	
os-caddy	 1.6.3

$ caddy -v
v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

Caddy installed via OPNsense plugin

OPNsense 24.7.3_1-amd64
FreeBSD 14.1-RELEASE-p3
OpenSSL 3.0.14

b. Command:

Caddy run via OPNSense Web GUI

c. Service/unit/compose file:

Not sure it is relevant, but Proxmox network setup

auto lo
iface lo inet loopback

auto enp3s0
iface enp3s0 inet manual

auto enx8cae4cddb32b
iface enx8cae4cddb32b inet manual

iface wlp4s0 inet manual

auto bond0
iface bond0 inet manual
        bond-slaves enp3s0 enx8cae4cddb32b
        bond-miimon 100
        bond-mode 802.3ad
        bond-xmit-hash-policy layer2+3

auto vmbr0
iface vmbr0 inet static
        bridge-ports bond0
        bridge-stp off
        bridge-fd 0
        bridge-vlan-aware yes
        bridge-vids 2-4094

auto vmbr0.10
iface vmbr0.10 inet static
        address 10.0.10.10/16
        gateway 10.0.10.1

d. My complete Caddy config:

Output from OPNSense Diagnostics (no other custom config, I pared it down the the minimum to test a working/not working one)

# DO NOT EDIT THIS FILE -- OPNsense auto-generated file

# caddy_user=root

# Global Options
{
	log {
		include http.log.access.f76fb375-99c1-465b-977b-4de404498305
		include http.log.access.d2de8d66-ef3d-43ac-a199-f2b872129c2a
		output net unixgram//var/run/caddy/log.sock {
		}
		format json {
			time_format rfc3339
		}
		level DEBUG
	}

	dynamic_dns {
		provider cloudflare <api-key>
		domains {
			maple.example.com @
			maple2.example.com @
		}
		versions ipv4
	}

	email acme.haiku428@passmail.com
	grace_period 10s
	import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Reverse Proxy Domain: "f76fb375-99c1-465b-977b-4de404498305"
maple.example.com {
	log f76fb375-99c1-465b-977b-4de404498305
	tls {
		issuer acme {
			dns cloudflare <api-key>
			resolvers 1.1.1.1
		}
	}

	handle {
		reverse_proxy 10.0.10.22:2283 {
		}
	}

	abort
}
# Reverse Proxy Domain: "d2de8d66-ef3d-43ac-a199-f2b872129c2a"
maple2.example.com {
	log d2de8d66-ef3d-43ac-a199-f2b872129c2a
	tls {
		issuer acme {
			dns cloudflare <api-key>
			resolvers 1.1.1.1
		}
	}

	handle {
		reverse_proxy 10.0.10.5:3000 {
		}
	}

	abort
}

import /usr/local/etc/caddy/caddy.d/*.conf

5. Links to relevant resources:

2

For sake of completeness, this is a fully working OPNsense-generated Caddyfile before I add the services on the Proxmox subnet. These continue to work fine as I add the Proxmox services as referenced above.

These three services are all on the same subnet/vlan as router; two on the OPNsense server itself and then the separate managed switch web UI.

DNS is handled locally on this one. I tested the both local dns and dynamic dns above to see if that was the issue.

# DO NOT EDIT THIS FILE -- OPNsense auto-generated file


# caddy_user=root

# Global Options
{
	log {
		include http.log.access.18add8d7-4e09-4680-bb55-d70e13f4f86e
		output net unixgram//var/run/caddy/log.sock {
		}
		format json {
			time_format rfc3339
		}
		level DEBUG
	}

	email acme.haiku428@passmail.com
	grace_period 10s
	import /usr/local/etc/caddy/caddy.d/*.global
}

# Reverse Proxy Configuration


# Reverse Proxy Domain: "18add8d7-4e09-4680-bb55-d70e13f4f86e"
*.example.com {
	log 18add8d7-4e09-4680-bb55-d70e13f4f86e
	tls {
		issuer acme {
			dns cloudflare <api-key>
			resolvers 1.1.1.1
		}
	}

	@4446e370-62de-47bf-b8dd-ac3aa672df12 {
		host router.example.com
	}
	handle @4446e370-62de-47bf-b8dd-ac3aa672df12 {
		@ded02ec4-be26-42eb-a9dd-2653f858414c {
			client_ip 192.168.0.0/16 10.0.0.0/8
		}

		handle @ded02ec4-be26-42eb-a9dd-2653f858414c {
			handle {
				reverse_proxy 192.168.1.1:8443 {
					transport http {
						tls_insecure_skip_verify
					}
				}
			}
		}

		abort
	}
	@b80a508d-866d-4534-976b-90fbd2eee64d {
		host switch.example.com
	}
	handle @b80a508d-866d-4534-976b-90fbd2eee64d {
		@ded02ec4-be26-42eb-a9dd-2653f858414c {
			client_ip 192.168.0.0/16 10.0.0.0/8
		}

		handle @ded02ec4-be26-42eb-a9dd-2653f858414c {
			handle {
				reverse_proxy 192.168.1.2 {
				}
			}
		}

		abort
	}
	@1c17fe94-3aa7-4494-a2bd-f45becae4632 {
		host adguard.example.com
	}
	handle @1c17fe94-3aa7-4494-a2bd-f45becae4632 {
		@26830483-5caf-4ef4-aa16-6108e3af4600 {
			client_ip 192.168.0.0/16 10.0.0.0/8 172.16.0.0/12
		}

		handle @26830483-5caf-4ef4-aa16-6108e3af4600 {
			handle {
				reverse_proxy 192.168.1.1:3000 {
				}
			}
		}

		abort
	}

	abort
}

import /usr/local/etc/caddy/caddy.d/*.conf
3

Ah!

I think I figured out what differentiates the working vs not for the Proxmox services:

  • Docker containers in LXC - os-caddy works
  • Services running in plain LXC - os-caddy not working

I believe the Proxmox container networking is all the same for both but maybe something different about those Docker LXCs I didn’t notice or the fact they are in Docker.

I will dig in this weekend but maybe that’ll help someone point me in the right direction.

ETA - Confirmed that running inside Docker fixes this. I have the same app running in the same LXC. It works fine running in Docker in the LXC, but outside just in the same LXC not using Docker I see requests for large files hanging (it does seem to only affect larger files).

I was able to get things working fine on another machine (also on a separate subnet/vlan), but without Proxmox. So it seems like neither Caddy nor the subnet itself is the issue, its something in that part of the network or Proxmox.

4

Glad you got it working in the end. These kind of issues are really hard to troubleshoot from the outside in the scope of a forum or reddit.

3 Likes
5

I setup another proxmox server still on a subnet/vlan and was also able to get that working. So its definitely something specific to how I set up the VLAN/Trunk on the switch to the original proxmox or the proxmox networking.

Good lesson learned on trying out simpler setups with fresh/separate devices first.

2 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.