How to get a true remote IP behind caddy reverse proxy

1. The problem I’m having:

The service behind caddy reverse proxy can NOT obtain true IP.

e.g.

  1. I have a server (IP 192.168.50.32), which is deploied the docker.
  2. The docker has two network mode: host and bridge(IP CIDR 172.17.0.0/20, Gateway 172.17.0.1)
  3. A Caddy docker is run using bridge (IP 172.17.0.5), which be set as a reverse proxy, 192.168.50.32:58443–>172.17.0.5:443
  4. A DDNS-GO docker is run using host, 192.168.50.32:9876
  5. A local network machine as terminal (IP 192.168.50.151)
  6. I expect machine access DDNS-GO via caddy, and DDNS-GO will obtain remote IP is 192.168.50.151, but now that is docker gateway IP 172.17.0.1

2. Error messages and/or full log output:

when I access ddns-go:

% curl -4vL https://app-ddnsgo.direct.do.main:58443/
*   Trying 192.168.50.32:58443...
* Connected to app-ddnsgo.direct.do.main (192.168.50.32) port 58443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.direct.do.main
*  start date: Jan 13 00:00:00 2024 GMT
*  expire date: Apr 12 23:59:59 2024 GMT
*  subjectAltName: host "app-ddnsgo.direct.do.main" matched cert''s "*.direct.noipto.host"
*  issuer: C=AT; O=ZeroSSL; CN=ZeroSSL ECC Domain Secure Site CA
*  SSL certificate verify ok.
* Using HTTP2, server supports multiplexing
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f9e2400b600)
> GET / HTTP/2
> Host: app-ddnsgo.direct.do.main:58443
> user-agent: curl/7.80.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!
< HTTP/2 403 
< alt-svc: h3=":443"; ma=2592000
< date: Mon, 15 Jan 2024 19:29:50 GMT
< server: Caddy
< content-length: 0
< 
* Connection #0 to host app-ddnsgo.direct.do.main left intact

This is caddy access log:

{
    "level": "error",
    "ts": 1705346990.384408,
    "logger": "http.log.access.log0",
    "msg": "handled request",
    "request": {
        "remote_ip": "172.17.0.1",
        "remote_port": "36576",
        "client_ip": "172.17.0.1",
        "proto": "HTTP/2.0",
        "method": "GET",
        "host": "app-ddnsgo.direct.do.main:58443",
        "uri": "/",
        "headers": {
            "User-Agent": [
                "curl/7.80.0"
            ],
            "Accept": [
                "*/*"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 772,
            "cipher_suite": 4865,
            "proto": "h2",
            "server_name": "app-ddnsgo.direct.do.main"
        }
    },
    "bytes_read": 0,
    "user_id": "",
    "duration": 0.00053049,
    "size": 0,
    "status": 403,
    "resp_headers": {
        "Server": [
            "Caddy"
        ],
        "Alt-Svc": [
            "h3=\":443\"; ma=2592000"
        ],
        "Date": [
            "Mon, 15 Jan 2024 19:29:50 GMT"
        ],
        "Content-Length": [
            "0"
        ]
    }
}

This is ddns-go access log

	2024/01/16 03:29:50 "Remote: 172.17.0.5:49442 ,Real-IP: 172.17.0.1:36576 ,Forwarded-For: 172.17.0.1:36576" prohibit acess from WAN

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

docker compose

b. Command:

caddy run --config /etc/caddy/Caddyfile

c. Service/unit/compose file:

services:
  caddy:
    container_name: Caddy
    image: caddy:latest
    restart: always
    network_mode: bridge
    ports:
      - "58443:443"
      - "58443:443/udp"
    volumes:
      - /etc/localtime:/etc/localtime
      - ./caddy/data:/data
      - ./caddy/config:/config
      - ./caddy/Caddyfile:/etc/caddy/Caddyfile

d. My complete Caddy config:

*.direct.do.main {
	@app-ddnsgo {
		host app-ddnsgo.direct.do.main
	}
	reverse_proxy @app-ddnsgo http://192.168.50.32:9876 {
		header_up Host {host}
		header_up X-Real-IP {remote}
		header_up X-Forwarded-For {remote}
        header_up X-Forwarded-Port {server_port}
        header_up X-Forwarded-Proto "http"
		trusted_proxies 172.17.0.0/20
	}
}

I try to comment Line 6-11 arbitrarily, but the reponse is NOT meet expectation.

5. Links to relevant resources:

None of this is useful in your case. You can remove all this.

Your problem is that Docker sometimes uses a userland-proxy, i.e. acts as a TCP-layer proxy in front of Caddy, which means the source IP is lost.

This isn’t a Caddy problem, but rather a Docker problem. This is a long-standing issue with Docker, for example Document how to get real remote client ip for service running in container · Issue #15086 · moby/moby · GitHub

Does this mean that if I install caddy without docker, the default reverse proxy can deliver true source IP, so I don’t need to extra caddyfile configuration?

Yes, running Caddy on the host (not in Docker) is one way to solve it.

I tried to running Caddy docker as network mode host, and modify caddy’s listen port, now it works!! The true source IP is shown in the service.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.