Trusted Proxies in OpenShift?

Help
1

1. The problem I’m having:

I currently having Caddy running in its own container hosted in our instance of OpenShift. It has a OpenShift route in front of it and then the whole cluster has an Azure WAF. I’m really struggling to get the X-Forwarded-For headers to work, even with trusted proxies set.

I’m monitoring the Caddy container logs and I can see the client IP address I want in the X-Forwarded-For headers, I just can’t seem to convince Caddy that the request is coming through trusted proxies. I’m sure it’s a me issue but can’t see any similar issues on the site!

2. Error messages and/or full log output:

{
    "level": "debug",
    "ts": 1710993658.8185718,
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "vigilant-server:3000",
    "duration": 0.008568237,
    "request": {
        "remote_ip": "172.19.4.5",
        "remote_port": "49426",
        "client_ip": "172.19.4.5",
        "proto": "HTTP/1.1",
        "method": "POST",
        "host": "vigilant",
        "uri": "/api/auth/refresh",
        "headers": {
            "X-Forwarded-Host": [
                "vigilant"
            ],
            "Content-Length": [
                "0"
            ],
            "Accept-Encoding": [
                "gzip, deflate, br"
            ],
            "X-Forwarded-Port": [
                "443"
            ],
            "X-Appgw-Trace-Id": [
                "6187e389e315137af17f17481f4b325c"
            ],
            "Sec-Fetch-Site": [
                "same-origin"
            ],
            "Sec-Fetch-Dest": [
                "empty"
            ],
            "Sec-Fetch-Mode": [
                "cors"
            ],
            "User-Agent": [
                "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.1.2 Safari/605.1.15"
            ],
            "X-Original-Host": [
                "vigilant"
            ],
            "Accept": [
                "*/*"
            ],
            "Accept-Language": [
                "en-GB,en;q=0.9"
            ],
            "X-Forwarded-For": [
                "37.60.X.Y:64393, 172.19.4.5"
            ],
            "X-Original-Url": [
                "/api/auth/refresh"
            ],
            "Content-Type": [
                "application/json; charset=UTF-8"
            ],
            "Referer": [
                "https://vigilant/?redirect=/profile"
            ],
            "Origin": [
                "https://vigilant"
            ],
            "X-Forwarded-Proto": [
                "https"
            ]
        },
        "tls": {
            "resumed": false,
            "version": 771,
            "cipher_suite": 49199,
            "proto": "",
            "server_name": "vigilant"
        }
    },
    "headers": {
        "Vary": [
            "Origin, Accept-Encoding"
        ],
        "Etag": [
            "W/\"45-r8VsLb3uGNauMtcyJtf8PznZRtg\""
        ],
        "Keep-Alive": [
            "timeout=5"
        ],
        "X-Powered-By": [
            "Express"
        ],
        "Access-Control-Allow-Credentials": [
            "true"
        ],
        "Content-Type": [
            "application/json; charset=utf-8"
        ],
        "Content-Length": [
            "69"
        ],
        "Date": [
            "Thu, 21 Mar 2024 04:00:58 GMT"
        ],
        "Connection": [
            "keep-alive"
        ]
    },
    "status": 403
}

As you can see the X-Forwarded-For header has the IP address I want but it’s not been moved over to the client_ip. It’s worth saying that the OpenShift route has TLS passthrough on.

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

a. System environment:

I am running the Alpine version of Caddy in your prebuilt Docker image on a Kubernetes cluster.

b. Command:

FROM docker.io/caddy:2.7-alpine

COPY waf/Caddyfile /etc/caddy/Caddyfile

USER 0
RUN adduser -HD -u 1001 -g caddy caddy
RUN chown -R caddy:caddy /usr/bin/caddy /etc/caddy

HEALTHCHECK CMD curl --fail http://localhost:3000/ || exit 1

USER caddy

EXPOSE 3000

CMD ["caddy", "run", "--config", "/etc/caddy/Caddyfile", "--adapter", "caddyfile"]```

c. My complete Caddy config:

{
  http_port 3001
  https_port 3000
  debug
  servers {
    trusted_proxies static private_ranges
  }
}

(good) {
}

:3001 {
  log
  reverse_proxy http://vigilant-client:3000 
}


:3000 {

  tls /etc/caddy/certs/tls.crt /etc/caddy/certs/tls.key

  encode gzip
  
  header +X-Frame-Options DENY
  header ?cache-control max-age=3600
  header +X-Content-Type-Options nosniff
  header +Strict-Transport-Security "max-age=31536000; includeSubDomains"
  header -X-Powered-By
  header +Content-Security-Policy "default-src 'none'; script-src 'self' 'unsafe-inline'; manifest-src 'self';connect-src 'self'; img-src 'self'; style-src 'self' 'unsafe-inline';base-uri 'self';form-action 'self'; font-src 'self';"
  #header +X-Content-Security-Policy default-src 'self'

  @internalapi {
    import good
    path /api/*
  }

  @internalui {
    import good
    path *
  }

  log { 
    output stdout
  }

   handle @internalapi {
    reverse_proxy https://vigilant-server:3000 {
      trusted_proxies private_ranges
      transport http {
        tls
        tls_insecure_skip_verify
      }
    }
  }


   handle @internalui {
    reverse_proxy http://vigilant-client:3000 
  }

  respond 403 {
    close
  }
}

This has had me stumped for about four weeks now so any help would be much appreciated!

2

OpenShift is Kubernetes, right? I don’t know much about it. That means you have a proxy in front of Caddy, I think.

Basically the problem is that Caddy sees the remote address as the IP of your OpenShift proxy, it doesn’t see the IP of the original client, because the remote address is the IP on the TCP packets.

Does OpenShift’s proxy actually set X-Forwarded-For? If it doesn’t, then configuring trusted_proxies won’t do anything because there’s no extra IP to grab from a header.

Or does it support PROXY protocol? That would be another way for the proxy to pass the original client IP to Caddy (but it’s something you’d need to turn on by configuring the proxy to add it to the connection).

Oh, read your post before looking at the logs… this looks fine though :thinking: maybe I don’t understand the problem?

One weird detail is XFF shouldn’t contain the port, that seems wrong. But not that important. Clearly Caddy is passing through XFF with the IP it sees as the remote address appended to it.

Oh, you mean this isn’t set as you expect.

Well, this is the proxy debug logs. That doesn’t really matter. Look at your access logs instead, it should be set correctly there.

I think it’s because OpenShift is sending the port, it shouldn’t be. The port isn’t relevant information.

3

Alright, I realized it was our mistake for not parsing off the port. OpenShift keeping the port is weird, but apparently it’s not unheard of, and it’s trivial to just remove it when determining the client IP.

4

Awesome! Thank you so much for your help. Whilst I wait for the next update I’ll see if I can rewrite the header to strip off the port.

1 Like
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.