Local reverse_proxy forced to external ip

Help
1

1. The problem I’m having:

Trying to route a new pgadmin docker container through my caddy reverse proxy like many others, only local reachable, microservices. But this time, caddy insists it has to be reachable from the outside, which results in a 403 error (intentionally), beaucse it should not be reachable from the internet.

2. Error messages and/or full log output:

image
image631×1144 55.8 KB

3. Caddy version:

v2.7.5

4. How I installed and ran Caddy:

docker-compose:

a. System environment:

Fedora release 37 (Thirty Seven

b. Command:

curl pg.home.<exxample>.de

*   Trying 90.186.20.654:443...
* Connected to pg.home.example.de (90.186.20.654) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS header, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS header, Finished (20):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.2 (OUT), TLS header, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=pg.home.example.de
*  start date: Nov 27 10:05:27 2023 GMT
*  expire date: Feb 25 10:05:26 2024 GMT
*  subjectAltName: host "pg.home.example.de" matched cert's "pg.home.example.de"
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* h2h3 [:method: GET]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: pg.home.example.de]
* h2h3 [user-agent: curl/7.85.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x557953e7d550)
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
> GET / HTTP/2
> Host: pg.home.example.de
> user-agent: curl/7.85.0
> accept: */*
>
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
* TLSv1.2 (OUT), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* TLSv1.2 (IN), TLS header, Supplemental data (23):
< HTTP/2 403
< alt-svc: h3=":443"; ma=2592000
< content-type: text/plain; charset=utf-8
< server: Caddy
< content-length: 13
< date: Thu, 30 Nov 2023 10:10:35 GMT
<
* TLSv1.2 (IN), TLS header, Supplemental data (23):
* Connection #0 to host pg.home.example.de left intact
Access Denied

c. Service/unit/compose file:

version: "3.9"
services:
  caddy:
    image: caddy:latest
    container_name: caddy
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
      - 443:443/udp
    volumes:
      - /home/example/homepage:/app
      - /home/example/docker/caddy/config:/config
      - /home/example/docker/caddy/caddy/data:/data
      - /home/example/docker/caddy/Caddyfile:/etc/caddy/Caddyfile
      - /home/example/docker/logs:/var/log/caddy
    environment:
      - DOMAIN=example.de
      - LCLDMN=home.example.de
      - TZ=Europe/Berlin
      - XDG_CONFIG_HOME=/config
      - XDG_DATA_HOME=/data
      - LOG_FILE=/data/access.log
    network_mode: bridge
    labels:
      - "com.centurylinklabs.watchtower.enable=false"

d. My complete Caddy config:

# global options   
{
	email admin@<example>.de
	on_demand_tls {
		interval 2m
		burst 5
	}
	servers {
		metrics
	}
	admin :2019
	debug
	log
}

# Snippetbox
snips.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.10.4:5000
}

#PGAdmin
pg.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}

	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:5433
}

# Duplicati
backup.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:8200
}

# dockerge
cocker.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:5022
}

# Homepage
dash.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:8832
}

# Prometheus
prometheus.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:9050
}

# OliveTin
auto.{$LCLDMN} {
	basicauth * {
		admin <password>
	}
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:1337
}

# Stirling
pdf.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.10.3:8080
}

# VSCode
code.{$LCLDMN} {
	basicauth * {
		admin <password>
	}
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:8443
}

# phpmyadmin
db.{$LCLDMN} {
	@non_local {
		not remote_ip 192.168.0.0/16
	}
	respond @non_local "Access Denied" 403
	reverse_proxy 192.168.0.28:5647
}

# external
www.{$DOMAIN} {
	redir https://{$DOMAIN}{uri}
}

# LinkStack
{$DOMAIN} {
	reverse_proxy 192.168.0.28:8153
	acme_server
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=31536000; includeSubDomains; preload
		X-Xss-Protection 1; mode=block
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy strict-origin-when-cross-origin
		Content-Security-Policy upgrade-insecure-requests
		Cache-Control public, max-age=15, must-revalidate
		Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none';       magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
	}
}

# Vaultwarden
pass.{$DOMAIN} {
	reverse_proxy 192.168.0.28:8010
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=31536000; includeSubDomains; preload
		X-Xss-Protection 1; mode=block
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy strict-origin-when-cross-origin
		Content-Security-Policy upgrade-insecure-requests
		Cache-Control public, max-age=15, must-revalidate
		Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none';       magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
	}
	reverse_proxy https://home.example.de {
		header_up Host {upstream_hostport}
	}
}

# Grafana
health.{$DOMAIN} {
	reverse_proxy 192.168.0.28:3000
	encode zstd gzip
	header {
		Strict-Transport-Security max-age=31536000; includeSubDomains; preload
		X-Xss-Protection 1; mode=block
		X-Content-Type-Options nosniff
		X-Frame-Options DENY
		Referrer-Policy strict-origin-when-cross-origin
		Content-Security-Policy upgrade-insecure-requests
		Cache-Control public, max-age=15, must-revalidate
		Feature-Policy "accelerometer 'none'; ambient-light-sensor 'none'; autoplay 'self'; camera 'none'; encrypted-media 'none'; fullscreen 'self'; geolocation 'none'; gyroscope 'none';       magnetometer 'none'; microphone 'none'; midi 'none'; payment 'none'; picture-in-picture *; speaker 'none'; sync-xhr 'none'; usb 'none'; vr 'none'"
	}
}

...

5. Links to relevant resources:

It works with other services like a charm. every other already in use microservice works inside my network but throws an 403 error. But the pgadmin ms is behaving particuarly like an external service and I cannot see why!

the caddy log doesn’t show anything of importance to this problem

2

I don’t understand the question. Can you try to re-explain the problem?

Show show your logs. Add the log directive in your site block (not in global options) to enable access logs.

Are you running this from a machine outside your network, or from inside your network?

No need to set this, this is already the default.

This doesn’t do anything for you, your site doesn’t enable On-Demand TLS.

3

Nevermind, i am fricking stupid and forgot to add pg.home.example.de to my local dns resolver (pihole) and wondered, why the query gets routed outside while the query is looking for a proper dns entry.

Sorry for the bother

Edit:
Thanks for the tipps :smiley:

4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.