Reverse proxy using specific outbound interface

Help
1

1. The problem I’m having:

I’m using Caddy in a Docker container and I’m trying to force Caddy to use a specific outbound interface when reverse proxying HTTP requests. I have a Home Assistant container running and I’m reverse proxying the web interface, and it needs a static IP address for the proxy. HA has a “trusted proxies” setting where you whitelist IP addresses that can set the X-Forwarded-For header.

I have a Docker network with static IP addressing and I need to make Caddy use that IP to access the Home Assistant container. Currently, it seems to just pick an interface at random when the container is started and stick to that interface. For example, if I give the Caddy container three different Docker networks, it uses a consistent interface after the container is started but which interface it chooses is a mystery to me.

The only relevant setting I can find is the bind directive, but that seems to limit the inbound interfaces, not outbound.

2. Error messages and/or full log output:

Caddy is running without errors. It’s proxying requests normally, I just want to force it to use a specified interface when proxying a certain subdomain in my Caddyfile.

3. Caddy version:

v2.6.4

4. How I installed and ran Caddy:

a. System environment:

Docker on Debian x64

b. Command:

docker compose up -d

c. Service/unit/compose file:

---
version: '3.7'
services:
  caddy:
    build:
      context: .
    restart: unless-stopped
    networks:
      heimdall:
        ipv4_address: 172.25.0.2
      freshrss:
      tautulli:
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - caddy-config:/config
      - caddy-data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile
    environment:
      - PROXY_DOMAIN=${PROXY_DOMAIN}
      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}

  homeassistant:
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - homeassistant:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    privileged: true
    network_mode: host

  heimdall:
    image: lscr.io/linuxserver/heimdall:latest
    restart: unless-stopped
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
    volumes:
      - heimdall:/config
    networks:
      - heimdall
    depends_on:
      - caddy

  freshrss:
    image: lscr.io/linuxserver/freshrss:latest
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
    volumes:
      - freshrss:/config
    restart: unless-stopped
    networks:
      - freshrss
    depends_on:
      - caddy

  tautulli:
    image: ghcr.io/tautulli/tautulli:latest
    restart: unless-stopped
    volumes:
      - tautulli:/config
    networks:
      - tautulli
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
    depends_on:
      - caddy

  plex:
    image: lscr.io/linuxserver/plex:latest
    network_mode: host
    volumes:
      - plex:/config
      - /mnt/data/Media/TV:/tv:ro
      - /mnt/data/Media/Movies:/movies:ro
      - /mnt/data/Media/Music:/music:ro
      - /mnt/data/Media/Other:/other:ro
    environment:
      - PUID=1000
      - PGID=1000
      - VERSION=docker
      - PLEX_CLAIM=${PLEX_CLAIM}
    restart: unless-stopped

networks:
  heimdall:
    ipam:
      config:                    # Set IP range so Caddy can get a static IP on
        - subnet: 172.25.0.0/24  # this network. Caddy should use this net to
  tautulli:                      # access Home Assistant and HA needs a static
  freshrss:                      # IP for the trusted proxy setting

volumes:
  caddy-config:
  caddy-data:           # Old names from previous compose. Going forward, these
    name: caddy-data    # should not be named volumes
    external: true
  plex:
    name: plex
    external: true
  heimdall:
  tautulli:
  homeassistant:
  freshrss:

d. My complete Caddy config:

{
	email redacted@example.com
	admin off
}

*.{$PROXY_DOMAIN} {
	tls {
		dns cloudflare {$CLOUDFLARE_API_TOKEN}
	}

	@homeassistant host homeassistant.{$PROXY_DOMAIN}
	handle @homeassistant {
		reverse_proxy http://docker.lan:8123
	}

	@heimdall host heimdall.{$PROXY_DOMAIN}
	handle @heimdall {
		reverse_proxy http://heimdall:80
	}

	@freshrss host rss.{$PROXY_DOMAIN}
	handle @freshrss {
		reverse_proxy http://freshrss:80
	}

	@tautulli host tautulli.{$PROXY_DOMAIN}
	handle @tautulli {
		reverse_proxy http://tautulli:8181
	}

	@plex host plex.{$PROXY_DOMAIN}
	handle @plex {
		reverse_proxy http://docker.lan:32400
	}
}

5. Links to relevant resources:

2

I don’t know of any way to configure Caddy to do this (but maybe someone else does), so let’s talk Docker.

One obvious way to do it is to simply only connect one network to the caddy container, thus eliminating the option to use the “wrong” network. Personally, I’ve set up a proxynet network that connects Caddy to all the containers that need to be proxied.

You can also set Docker networks to internal: true, which would prevent them being used to communicate with the outside world (including the host network, AFAIK). If you set all but one of the networks connected to the caddy container to internal, that should ensure there’s only one network for it to use to connect to Home Assistant. This may require connecting additional networks to other hosts if they do need external connectivity.

As a third option, does Home Assistant really need to be using network_mode: host? If not, all you can simply ensure there’s only a single network connected to both the homeassistant and caddy containers (and reconfigure Caddy to connect to homeassistant instead of docker.lan).

1 Like
3

Interesting point about the internal/external Docker network setting. Seems tedious but might be a solution.

I know I could put all my containers in one network and it would work fine, but I’m trying to segment them as much as possible. I know my home network isn’t as secure as it could be but I’m trying to improve my security. Maybe using segmented Docker networks doesn’t provide a whole lot of security, but I wanted to err on the side of caution. I might give up on that notion though.

And no, Home Assistant doesn’t need the network mode: host option, but since there are so many different things it could connect with (and any number of ports that the container would need opened), I took the lazy route. See this thread on their forums. I could try to just manually open ports on a bridge network and see if anything breaks.

Thanks for your advice!

4

Does HASS not allow you to set a subnet as a trusted proxy?

List of trusted proxies, consisting of IP addresses or networks, that are allowed to set the X-Forwarded-For header.

—https://www.home-assistant.io/integrations/http/#trusted_proxies, with emphasis added

You could set 172.16.0.0/12 to set that entire private IP address space (including all the Docker IP space) as trusted, theoretically.

1 Like
5

Yeah, that’s another solution for sure. I’d like to be a bit less permissive but that’s still a good option. I assumed Caddy had this functionality but if not, then it seems I’ve got plenty of ideas for workarounds. Thanks for the help.

6

Just in case someone else down the line has the same issue of wanting Caddy in Docker to use a specific outbound interface:

Since Caddy currently doesn’t have that functionality, I decided to take one of the suggestions here and designate one Docker network as an “external access” network and make all the other ones internal (since they’re proxying to other Docker containers). This ensures that whenever Caddy wants to access an external resource (i.e., anything that isn’t a Docker container with an internal network attached), it uses this external access network.

Here’s my docker-compose.yml:

---
version: '3.7'

networks:
  externalnetwork:               # Set IP range so Caddy can get a static IP on
    ipam:                        # this network. Caddy will use this net to
      config:                    # access Home Assistant because HA needs a static
        - subnet: 172.25.0.0/24  # IP for the trusted proxy setting
  internalapp:
    internal: true

services:
  caddy:
    image: <my-custom-image-with-cloudflare-plugin>
    restart: unless-stopped
    networks:
      externalnetwork:            # Or whatever you want the static IP to be, so
        ipv4_address: 172.25.0.2  # long as it's within the subnet you set above
      internalnetwork:
    ports:
      - '80:80'
      - '443:443'
    volumes:
      - caddy-config:/config
      - caddy-data:/data
      - ./Caddyfile:/etc/caddy/Caddyfile
    environment:
      - PROXY_DOMAIN=${PROXY_DOMAIN}
      - CLOUDFLARE_API_TOKEN=${CLOUDFLARE_API_TOKEN}

  homeassistant:
    image: "ghcr.io/home-assistant/home-assistant:stable"
    volumes:
      - homeassistant:/config
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    privileged: true
    network_mode: host  # Since HA needs services that use multicast packets, you're kinda
                        # stuck with "network_mode: host" or using macvlan. I chose the former

  internalapp:
    image: lscr.io/linuxserver/freshrss:latest
    environment:
      - PUID=1000
      - PGID=1000
      - TZ=America/Chicago
    volumes:
      - freshrss:/config
    restart: unless-stopped
    networks:
      - internalapp1  # This guy will use an internal network, so we don't worry about IP addressing
    depends_on:
      - caddy

volumes:
  caddy-config:
  caddy-data:
  homeassistant:
  freshrss:

And my Caddyfile looks like:

{
	email admin@example.com
	admin off
}

*.{$PROXY_DOMAIN} {
	tls {
		dns cloudflare {$CLOUDFLARE_API_TOKEN}
	}

	@internalapp host internalapp.{$PROXY_DOMAIN}
	handle @internalapp {
		reverse_proxy http://containername:1234  # This uses Docker DNS to resolve to an internal network
	}											 # We don't care what IP it uses

	@homeassistant host homeassistant.{$PROXY_DOMAIN}
	handle @homeassistant {
		reverse_proxy http://hostname.tld:8123  # This uses external DNS to resolve to an external network
	}											# Caddy will use the static IP for this connection
}

Thanks to @fvbommel for the idea.

1 Like
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.