Can't get client IP on node/express website behind caddy reverse proxy

Help
1

1. The problem I’m having:

No matter what I do I can’t get the client IP on my web server. I get the server IP (10.1.10.XXX) as the client IP and the X-Forwarded-For is the 172.18.0.X IP from the internal docker network. I am running a nodejs express website directly on the windows machine and have previously had it behind a commercial grade reverse proxy for a while with no issues. I really need the client IP as it is used for some blacklisting of IPs and countries. I have also setup a development server on a separate machine with the same results.

2. Error messages and/or full log output:

Here is some text that I get from my test page that outputs IP information and all headers from the request.

Source IP: 10.1.10.136, Source Port : 49500
Node Version: v20.17.0
All Headers: {
  "host": "development.mydomain.com",
  "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
  "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
  "accept-encoding": "gzip, deflate, br, zstd",
  "accept-language": "en-US,en;q=0.9",
  "cache-control": "max-age=0",
  "if-none-match": "W/\"ac4-Xx69upfr1BZhYNSCqln9hLeJmXs\"",
  "priority": "u=0, i",
  "sec-ch-ua": "\"Google Chrome\";v=\"129\", \"Not=A?Brand\";v=\"8\", \"Chromium\";v=\"129\"",
  "sec-ch-ua-mobile": "?0",
  "sec-ch-ua-platform": "\"Windows\"",
  "sec-fetch-dest": "document",
  "sec-fetch-mode": "navigate",
  "sec-fetch-site": "none",
  "sec-fetch-user": "?1",
  "upgrade-insecure-requests": "1",
  "x-forwarded-for": "172.18.0.1:40768",
  "x-forwarded-host": "development.mydomain.com:4430",
  "x-forwarded-port": "4430",
  "x-forwarded-proto": "https",
  "x-real-ip": "172.18.0.1:40768"
}

3. Caddy version:

V2.8.4

4. How I installed and ran Caddy:

I added caddy to my docker-compose file manually

  caddy:
    image: caddy:latest
    restart: unless-stopped
    ports:
      - "8000:8000"
      - "4430:4430"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./certs/privkey.pem:/etc/caddy/privkey.pem
      - ./certs/cert.pem:/etc/caddy/cert.pem
    depends_on:
      - immich-server

a. System environment:

Running on docker-compose under windows. The nodejs/express web server is running on the same box but directly under windows.

b. Command:

docker compose pull && docker compose up -d

c. Service/unit/compose file:

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 8283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
      start_interval: 30s
      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

  caddy:
    image: caddy:latest
    restart: unless-stopped
    ports:
      - "8000:8000"
      - "4430:4430"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./certs/privkey.pem:/etc/caddy/privkey.pem
      - ./certs/cert.pem:/etc/caddy/cert.pem
    depends_on:
      - immich-server


volumes:
  model-cache:

d. My complete Caddy config:

## General Options
{
  ## Adjust ports to your needs
  http_port 8000
  https_port 4430 
}


## Your wildcard domain
*.mydomain.com {
  
  ## Your wildcard certificate and its key
  tls /etc/caddy/cert.pem /etc/caddy/privkey.pem
  
  ## Immich Container
  @immich host immich.mydomain.com
  handle @immich {   
    reverse_proxy immich_server:3001
    #reverse_proxy http://10.1.10.136:8283
  }
  @openwebui host openwebui.mydomain.com
  handle @openwebui {
    reverse_proxy http://10.1.10.136:3000
  }
  @webmail host webmail.mydomain.com
  handle @webmail {
    reverse_proxy https://10.1.10.136:8443 {
      transport http {
        tls_insecure_skip_verify
      }
    }
  }
  @development host development.mydomain.com
  handle @development {
    reverse_proxy https://10.1.10.141:443 {
      header_up Host {http.request.host}
      header_up X-Real-IP {http.request.remote}
      header_up X-Forwarded-For {http.request.remote}
      header_up X-Forwarded-Port {http.request.port}
      header_up X-Forwarded-Proto {http.request.scheme}
      transport http {
        tls_insecure_skip_verify
      }
    }
  }

  ## Fallback for otherwise unhandled domains
  handle {
    reverse_proxy https://10.1.10.136:443 {
      header_up X-Real-IP {remote_host}
      transport http {
        tls_insecure_skip_verify
      }
    }
  }
}

5. Links to relevant resources:

N/A

2

Remove all this. It’s not useful. In fact, it messes up Caddy’s defaults. See reverse_proxy (Caddyfile directive) — Caddy Documentation

Since you’re running Caddy in Docker, your Docker might have enabled the userland proxy which causes TCP connections to get proxied by it, meaning the TCP connections that reach Caddy look like they come from Docker and not from the original client. You can turn this off via Docker daemon config.

1 Like
3

Turned off the userland-proxy and restarted docker. No difference.

4

By “restarted docker” do you mean the Docker daemon, or just your containers? You need to restart the daemon after making config changes to it. You may also need to recreate your containers (docker compose down) and bring them back up to reset their network settings.

1 Like
5

Restarted docker and then all the containers were restarted.

6

Also ran docker compose down then back up. The only thing that has changed is it seems much faster for its initial load of the apps running in the containers. Still same issue with the IP addresses. I ran with userland-proxy:false and then tried with that and iptables:true neither made any difference except like I said it appears the first load of a site seems much faster.

I definitely feel it is docker and network related though just don’t know how to get the real client IP across docker and into caddy then into my apps.

7

Would it make sense to try and run caddy on bare metal?

8

You certainly could.

I don’t understand why Docker is still messing with the connection if the userland proxy is off. There might be some other mechanism it uses that I don’t know about.

1 Like