Reverse-Proxy chaining: VPS -> Local server -> Service. Trusted Proxies?

IngwiePhoenix · December 13, 2022, 3:43pm

1. Output of `caddy version`:

root@birb:/srv# caddy version
v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

root@FriendlyWrt:~# caddy version
v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

VPS: SystemD
OpenWrt: /etc/rc.local + caddy reload/stop/start/...

a. System environment:

VPS:

Debian 11
OpenWrt:

# cat /etc/openwrt_release
DISTRIB_ID='OpenWrt'
DISTRIB_RELEASE='22.03.2'
DISTRIB_REVISION='r19803-9a599fee93'
DISTRIB_TARGET='rockchip/armv8'
DISTRIB_ARCH='aarch64_generic'
DISTRIB_DESCRIPTION='OpenWrt 22.03.2 r19803-9a599fee93'
DISTRIB_TAINTS='busybox'

b. Command:

Not applicable here... handled by init systems.

c. Service/unit/compose file:

VPS: Default systemd unit
OpenWrt:

# /etc/rc.local:
/usr/bin/caddy start --resume --config /srv/Caddyfile

d. My complete Caddy config:

Ok, this is where it becomes a little messy.

First, my VPS’ config:

birb.it {
        root * /srv/birb.it
        file_server
}
# ... cutting out other unrelated domains ...
#*.birb.it {
#  # Home server link
#  reverse_proxy * 192.168.222.11:80 {
#    transport http {
#      # Add to nanopi: trusted_proxies 123.123.123.123
#      # maybe? tls_trusted_ca_certs <pem_files...>
#      compression on
#    }
#  }
#}

And my OpenWrt one:

# Globals
{
        log {
                level info
                output file /var/log/caddy.log {
                        roll_size 10mb
                        roll_keep 10
                        roll_keep_for 720h
                }
        }
        trusted_proxies 192.168.222.0/24 # <- Error, invalid!
}

(php) {
        php_fastcgi unix//var/run/php8-fpm.sock {
                capture_stderr true
        }
}

:80 {
        # Set this path to your site's directory.
        root * /srv/default

        # Enable the static file server.
        file_server

        # Another common task is to set up a reverse proxy:
        # reverse_proxy localhost:8080

        # Or serve a PHP site through php-fpm:
        # php_fastcgi localhost:9000
}

# Basic ping-pong
echo.birb.it:80 {
        respond "hello"
}

# LuCi
router.birb.it:80 {
        reverse_proxy * localhost:8080
}

# Home Assistant
hass.birb.it:80 {
        reverse_proxy * localhost:8123
}

# TVHeadend
tvh.birb.it:80 {
        reverse_proxy * localhost:9981 {
                header_up X-Real-IP {remote_host}
                header_up X-Forwarded-For {remote_host}
        }
}

# Jellyfin
video.birb.it:80 {
        reverse_proxy * localhost:8096
}

# Navidrome
audio.birb.it:80 {
        reverse_proxy * localhost:4533
}

# Photoprism...?
# photo.birb.it:80 { reverse_proxy * localhost:x }

# NextCloud
cloud.birb.it:80 {
        root * /sdcard/srv/nextcloud/
        import php
        file_server
        redir /.well-known/carddav /remote.php/dav 301
        redir /.well-known/caldav /remote.php/dav 301
        redir /.well-known/webfinger /index.php/.well-known/webfinger 301
        redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301
        @forbidden {
                path /.htaccess /data/* /config/* /db_structure /.xml /README /3rdparty/* /lib/* /templates/* /occ /console.php
        }
        respond @forbidden 404
}

# Monica
monica.birb.it:80 {
        root * /sdcard/srv/monica/public
        import php
        file_server
}

# Grocy
grocy.birb.it:80 {
        root * /sdcard/srv/grocy/public
        import php
        file_server
}

# Paperless / Paperspace
# paper.birb.it:80 { reverse_proxy * localhost:x }

tubesync.birb.it:80 {
        reverse_proxy * localhost:4848
}

neko.birb.it:80 {
        reverse_proxy * localhost:8989
}

rclone.birb.it:80 {
        root * /www/rclone-webui-react/
        file_server
}

3. The problem I’m having:

From my local server itself, OpenWrt, it looks fine (i abused my Grocy host for this test ^^')

root@FriendlyWrt:/sdcard/srv/grocy/public# curl grocy.birb.it/hdrs.php
Array
(
    [TEMP] => /tmp
    [TMPDIR] => /tmp
    [TMP] => /tmp
    [PATH] => /usr/local/bin:/usr/bin:/bin
    [HOSTNAME] =>
    [USER] => www
    [HOME] => /home/www
    [REQUEST_URI] => /hdrs.php
    [SERVER_PROTOCOL] => HTTP/1.1
    [GATEWAY_INTERFACE] => CGI/1.1
    [HTTP_X_FORWARDED_HOST] => grocy.birb.it
    [SERVER_PORT] => 80
    [REQUEST_METHOD] => GET
    [REMOTE_USER] =>
    [HTTP_ACCEPT] => */*
    [REMOTE_PORT] => 54042
    [REMOTE_ADDR] => 192.168.2.1
    [QUERY_STRING] =>
    [CONTENT_LENGTH] => 0
    [HTTP_X_FORWARDED_PROTO] => http
    [HTTP_HOST] => grocy.birb.it
    [REQUEST_SCHEME] => http
    [REMOTE_IDENT] =>
    [HTTP_USER_AGENT] => curl/7.86.0
    [PATH_INFO] =>
    [SCRIPT_FILENAME] => /sdcard/srv/grocy/public/hdrs.php
    [DOCUMENT_ROOT] => /sdcard/srv/grocy/public
    [HTTP_X_FORWARDED_FOR] => 192.168.2.1
    [DOCUMENT_URI] => /hdrs.php
    [SERVER_SOFTWARE] => Caddy/v2.6.2
    [SERVER_NAME] => grocy.birb.it
    [REMOTE_HOST] => 192.168.2.1
    [SCRIPT_NAME] => /hdrs.php
    [CONTENT_TYPE] =>
    [AUTH_TYPE] =>
    [FCGI_ROLE] => RESPONDER
    [PHP_SELF] => /hdrs.php
    [REQUEST_TIME_FLOAT] => 1670944509.83
    [REQUEST_TIME] => 1670944509
    [argv] => Array
        (
        )

    [argc] => 0
)

Now, from my VPS via it’s VPN link (192.168.222.0/24):

root@birb:/srv# curl -H "Host: grocy.birb.it" 192.168.222.11/hdrs.php
Array
(
    [TEMP] => /tmp
    [TMPDIR] => /tmp
    [TMP] => /tmp
    [PATH] => /usr/local/bin:/usr/bin:/bin
    [HOSTNAME] =>
    [USER] => www
    [HOME] => /home/www
    [HTTP_X_FORWARDED_FOR] => 192.168.222.10
    [SCRIPT_FILENAME] => /sdcard/srv/grocy/public/hdrs.php
    [HTTP_HOST] => grocy.birb.it
    [REQUEST_URI] => /hdrs.php
    [DOCUMENT_URI] => /hdrs.php
    [SERVER_SOFTWARE] => Caddy/v2.6.2
    [QUERY_STRING] =>
    [CONTENT_TYPE] =>
    [HTTP_X_FORWARDED_PROTO] => http
    [SERVER_PORT] => 80
    [REMOTE_USER] =>
    [HTTP_ACCEPT] => */*
    [REQUEST_SCHEME] => http
    [SCRIPT_NAME] => /hdrs.php
    [SERVER_PROTOCOL] => HTTP/1.1
    [REMOTE_PORT] => 42306
    [PATH_INFO] =>
    [CONTENT_LENGTH] => 0
    [HTTP_USER_AGENT] => curl/7.74.0
    [REQUEST_METHOD] => GET
    [GATEWAY_INTERFACE] => CGI/1.1
    [SERVER_NAME] => grocy.birb.it
    [REMOTE_HOST] => 192.168.222.10
    [REMOTE_ADDR] => 192.168.222.10
    [REMOTE_IDENT] =>
    [AUTH_TYPE] =>
    [HTTP_X_FORWARDED_HOST] => grocy.birb.it
    [DOCUMENT_ROOT] => /sdcard/srv/grocy/public
    [FCGI_ROLE] => RESPONDER
    [PHP_SELF] => /hdrs.php
    [REQUEST_TIME_FLOAT] => 1670945586.72
    [REQUEST_TIME] => 1670945586
    [argv] => Array
        (
        )

    [argc] => 0
)

And finally, I added a tiny reverse-proxy block for grocy.birb.it. Accessing this from a totally different server reveals:

As you can see, the X-Forwarded-For header is not set properly - its set to my VPS’, but if you ping drachennetz.com, you can tell that this isn’t exactly correct…

This is the snippet I inserted for the last test:

grocy.birb.it {
  reverse_proxy * 192.168.222.11:80
}

4. Error messages and/or full log output:

No error messages, only misaligned headers.

5. What I already tried:

I tried to add trusted_proxies 192.168.222.10 to my root config, but that was not correct, caddy validate rejected that. Then I looked at the directive itself in the docs,m and there it is only mentioned as part of the reverse_proxy block. So… Do I have to add that to /every/ single reverse proxy setup?..

Thank you!

PS.: My next step is to make VPS and OpenWRT share the same config directory with the latter not auto-renewing and leaving that to the VPS and simply pulling the key bundles from there to allow for HTTPS on the local network with the correct SNI. The router is configured to statically assign *.birb.it to itself. The idea: Route to the router when at home, and go through a reverse proxy/VPN chain when away and only expose select services (Grocy would be one). Once that is working, TOTP and such to harden those exposed services properly. But, for now, I need to get the basics working - and having the correct IP in requests is kinda important… ^^’

Thank you and kind regards,
Ingwie!

6. Links to relevant resources:

None

francislavoie · December 13, 2022, 6:32pm

For now, yes.

There is a PR in the works to make it configurable via global options, but it’s not merged yet. I haven’t had the time to finish it up yet, but I will soon.

github.com/caddyserver/caddy

caddyhttp: Add server-level `trusted_proxies` config

caddyserver:master ← caddyserver:global-trusted-proxies

opened 03:29AM - 01 Oct 22 UTC

francislavoie

+117 -23

Partial work for #4924 as per my plan in https://github.com/caddyserver/caddy/is…sues/4924#issuecomment-1206477962 We've had requests for defining `trusted_proxies` globally, so it doesn't need to be configured for each `reverse_proxy`. This does that. But also, this opens the door for more places we can use trusted proxies, such as in #4924 which asks for getting the real client IP. We can do that by looking at the XFF header _if the proxy is trusted_. That part isn't done in this PR, _edit:_ this is in #5104! Also we might want to change how `remote_ip forwarded` works, possibly making it not trust forwarded by default (which would be a breaking change... but eh, it's known to be insecure right now). ~~And maybe we should add `client_ip` to the logs.~~ See #5104 for the logs part.

IngwiePhoenix · December 13, 2022, 11:31pm

Oh boy. Welp, let the copy-pasta begin! xD

Just to clarify:

# On VPS:
domain {
  reverse_proxy * router_via_vpn
}

# On router:
domain {
  trusted_proxies 192.168.222.0/24
  ... other flags ...
}

Yes?

francislavoie · December 13, 2022, 11:42pm

No, trusted_proxies is an option that goes inside of reverse_proxy and/or php_fastcgi.

system · January 12, 2023, 3:43pm

This topic was automatically closed after 30 days. New replies are no longer allowed.