Forward to service under another caddy setup

1. The problem I’m having:

Hello. To avoid any XY problems, i have fairly odd setup. To access services on my local network, i set up AWS instance that acts as proxy. It is normal ubuntu server with wireguard server on it listening on specific port. I connect to it with my homeserver as a client. On the server side (the AWS instance side), i set up some iptables magic to forward the traffic to and from my local
home network. Recently i wanted to deploy Keycloak so everything would have a login and be potentially more secure with better and centralized control. So i spun up new VM (besides the one where i host all of my services in docker environment with the main caddy instance and also is the wireguard client i talked about above), typical Ubuntu Server 24.04 LTS, installed docker and spun up Keycloak inside the docker. The setup looked promising so i decided to wrap it behind Caddy.

I realised the first issue. How can i set up caddy on the instance which is not connected with the AWS proxy instance? So i made new client, connected it via wireguard to AWS instance and found out i simply cannot route the traffic based on the request because the outgoing port is already taken by the first, main, client. So i thought a bit and decided to spin up new caddy instance on the keycloak VM so i could hide it behind TLS in my local network to have somethig like keycloak.internal.my-domain.com and access it locally. I did it and it works nicely. Alright, but i want it to have publicly available so i can wrap keycloak around my publicly available services. So i added it into configuration like:

keycloak.my-domain.com {
  reverse_proxy keycloak.internal.my-domain.com
}

and added it into my dns entry (i entered the AWS instance IP as always). I went to that dns name and got ERR_TOO_MANY_REDIRECTS which is, i guess, exactly what i should get. My question is, how can i forward it so the communication would be secure?

2. Error messages and/or full log output:

ERR_TOO_MANY_REDIRECTS

but the caddy logs looks like:

2025/02/20 19:17:03.211	DEBUG	http.handlers.reverse_proxy	upstream roundtrip	{"upstream": "keycloak.internal.my-domain.com:80", "duration": 0.007829142, "request": {"remote_ip": "10.0.0.1", "remote_port": "14468", "client_ip": "10.0.0.1", "proto": "HTTP/2.0", "method": "GET", "host": "keycloak.my-domain.com", "uri": "/", "headers": {"Priority": ["u=0, i"], "Cache-Control": ["max-age=0"], "Accept-Language": ["sk-SK,sk;q=0.9,en-US;q=0.8,en;q=0.7,cs;q=0.6"], "X-Forwarded-Proto": ["https"], "Sec-Ch-Ua": ["\"Not A(Brand\";v=\"8\", \"Chromium\";v=\"132\", \"Google Chrome\";v=\"132\""], "Sec-Ch-Ua-Mobile": ["?0"], "X-Forwarded-Host": ["keycloak.my-domain.com"], "Dnt": ["1"], "User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/132.0.0.0 Safari/537.36"], "Sec-Fetch-Mode": ["navigate"], "Accept-Encoding": ["gzip, deflate, br, zstd"], "Sec-Ch-Ua-Platform": ["\"Windows\""], "Cookie": ["REDACTED"], "Upgrade-Insecure-Requests": ["1"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"], "Sec-Fetch-Dest": ["document"], "Sec-Fetch-User": ["?1"], "Sec-Fetch-Site": ["cross-site"], "X-Forwarded-For": ["10.0.0.1"]}, "tls": {"resumed": true, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "keycloak.my-domain.com"}}, "headers": {"Location": ["https://keycloak.my-domain.com/"], "Server": ["Caddy"], "Date": ["Thu, 20 Feb 2025 19:17:03 GMT"], "Content-Length": ["0"]}, "status": 308}

3. Caddy version:

docker image: iarekylew00t/caddy-cloudflare:2.8.4

4. How I installed and ran Caddy:

a. System environment:

Ubuntu Server 22.04 LTS (the main VM with all the services)

  • Docker version 26.1.4, build 5650f9b

Ubuntu Server 24.04 LTS (the keycloak machine)

  • Docker version 28.0.0, build f9ced58

b. Command:

No changes

c. Service/unit/compose file:

main machine’s:

version: '3.3'

services:
  caddy:
    dns:
      - 192.168.96.4 #The IP to my PiHole local DNS server within the same docker network
    image: iarekylew00t/caddy-cloudflare:2.8.4
    stdin_open: true # docker run -i
    tty: true        # docker run -t
    restart: unless-stopped
    container_name: caddy
    ports:
      - 80:80
      - 443:443
    volumes:
      - /home/docker/docker_caddy/caddy:/etc/caddy
      - /home/docker/docker_caddy/site:/srv
      - /home/docker/docker_caddy/caddy_data:/data
      - /home/docker/docker_caddy/caddy_config:/config
    environment:
      CLOUDFLARE_API_TOKEN: "REDACTED"
    networks:
      - caddy
networks:
  caddy:
    name: proxy

PS: It looks the same on the keycloak machine

d. My complete Caddy config:

{
  debug
  email my-email@gmail.com
}

# btw, this entry works nicely, just to comment it...
immich.my-domain.com {
  reverse_proxy 192.168.1.13:2283
}

keycloak.my-domain.com {
  reverse_proxy keycloak.internal.my-domain.com
}

*.internal.my-domain.com {
  tls {
    dns cloudflare {env.CLOUDFLARE_API_TOKEN}
  }

  @services host services.internal.my-domain.com
  handle @services {
#    @headerfilter {
#      not header Cookie *secret=lala*
#    }
#    respond @headerfilter 403
    reverse_proxy 192.168.1.13:1111
  }

  @portainer host portainer.internal.my-domain.com
  handle @portainer {
 #   @headerfilter {
 #     not header Cookie *secret=lala*
 #   }
 #   respond @headerfilter 403
    reverse_proxy 192.168.1.13:9000
  }

  @homeassistant host homeassistant.internal.my-domain.com
  handle @homeassistant {
    reverse_proxy 192.168.1.172:8123
  }

  @actualbudget host actualbudget.internal.my-domain.com
  handle @actualbudget {
    reverse_proxy 192.168.1.13:4170
  }

  @pihole host pihole.internal.my-domain.com
  handle @pihole {
    reverse_proxy 192.168.1.13:4140
  }

  @ntfy host ntfy.internal.my-domain.com
  handle @ntfy {
    reverse_proxy 192.168.1.13:4110
    @httpget {
      protocol http
      method GET
      path_regexp ^/([-_a-z0-9]{0,64}$|docs/|static/)
    }
    redir @httpget https://{host}{uri}
  }
}

5. Links to relevant resources:

None

The hostname keycloak.internal.my-domain.com is part of *.internal.my-domain.com

This means it’s served by Caddy, which enforces HTTPS by Caddy issuing HTTP 308 to the HTTPS location. Change the upstream thusly:

keycloak.my-domain.com {
-  reverse_proxy keycloak.internal.my-domain.com
+  reverse_proxy https://keycloak.internal.my-domain.com {
+    header_up Host {upstream_hostport}
+  }
}

This tells Caddy to connect to upstream via HTTPS

2 Likes

Amazing job!

Thank you very much! I didn’t know it is enough to change the Host header so it would work properly! In the host port is defined the final host and port right? If that doesn’t change, the Host header would have final destination of keycloak.my-domain.com and not the internal host and that is why it would throw too many redirects? It is just a bonus question to widen my knowledge. :slight_smile:

Update: I probably spoke too soon. I am getting error in Keycloak logs:

2025-02-22 09:33:41,171 WARN [org.keycloak.cookie.DefaultCookieProvider] (executor-thread-94) Non-secure context detected; cookies are not secured, and will not be available in cross-origin POST requests

Update2: I found this issue on Keycloak’s github together with this warning. Do you have any experience with setting these parameters on Caddy’s end?

That’s partially why. You’re missing that your older config used to connect to upstream on http (no s), but the change also adds connection over https (with S). In your older config, the upstream Caddy responds with a redirect to the HTTPS port.

Caddy sets those headers by default. You just have to configure Keycloak accordingly.

From what I understood from your message and hours of searching, I need to define in caddy to set X-Forwarded-* headers so Keycloak know how to handle the requests?

I might also need to set PROXY_ADDRESS_FORWARDING variable on Keycloak’s ene to true.

PS: I can’t believe there is noone that really published their Caddyfile for Keycloak.

No, I said Caddy sets them by default.

Maybe? That’s a Keycloack question, but I think this is correct.

1 Like

Yes, you were right. I played with it a bit and found out i need to have to specify command parameter on keycloak’s end: --proxy-headers xforwarded, so now the configuration looks like:

keycloak docker-compose:

version: '3'
services:
  postgresql:
    image: postgres:16
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - '/home/guard/docker/keycloak/postgresql_data:/var/lib/postgresql/data'
    networks:
      keycloak:

  keycloak:
    image: quay.io/keycloak/keycloak:26.1
    restart: always
    command: start --hostname https://keycloak.internal.my-domain.com --proxy-headers xforwarded --http-enabled true
    depends_on:
      - postgresql
    environment:
      - KC_PROXY_ADDRESS_FORWARDING=true
      - KC_HTTP_ENABLED=true
      - KC_DB=postgres
      - KC_DB_USERNAME=${POSTGRES_USER}
      - KC_DB_PASSWORD=${POSTGRES_PASSWORD}
      - KC_DB_URL_HOST=postgresql
      - KC_DB_URL_PORT=5432
      - KC_DB_URL_DATABASE=keycloak
      - KEYCLOAK_ADMIN=${KEYCLOAK_ADMIN}
      - KEYCLOAK_ADMIN_PASSWORD=${KEYCLOAK_ADMIN_PASSWORD}

    networks:
      proxy:
      keycloak:

networks:
  proxy:
    external: true
  keycloak:

Caddyfile:

{
  debug
  email email@gmail.com
}

*.internal.dashrave.eu {
  tls {
    dns cloudflare {env.CLOUDFLARE_API_TOKEN}
  }

  @keycloak host keycloak.internal.my-domain.com
  handle @keycloak {
    reverse_proxy keycloak-keycloak-1:8080 {
      header_up Host {upstream_hostport}
      header_up X-Forwarded-For {remote}
      header_up X-Forwarded-Proto {scheme}
    }
  }
}

(I probably didn’t need to header up the x-forwarded-* but it works with them).

Now i have different issue. As it worked nicely and the warn went away, i wanted to publish it so i can start using it without being on my local network. I went ahead and configured the main caddy (on the VM where are all the services - not the keycloak VM) like this:

keycloak.dashrave.eu {
  reverse_proxy https://keycloak.internal.dashrave.eu {
    header_up Host {upstream_hostport}
    header_up X-Forwarded-For {remote}
    header_up X-Forwarded-Proto {scheme}
  }
}

Like you recommended me in the first reply you wrote. I added extra those two headers so Keycloak knows how to handle the requests / communication. I went to the website keycloak.my-domain.com, it loaded just fine, and logged in, into my newly created realm (not the master one) with the user i created. Note that this login worked while was on keycloak.internal.my-domain.com URL. But instead, it went into infinite loading screen with chrome log:

An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing.Understand this warningAI
/realms/master/protocol/openid-connect/3p-cookies/step2.html:1

An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing.Understand this warningAI
login-status-iframe.html:1

An iframe which has both allow-scripts and allow-same-origin for its sandbox attribute can escape its sandboxing.Understand this warningAI
/realms/master/protocol/openid-connect/login-status-iframe.html/init?client_id=account-console&origin=https%3A%2F%2Fkeycloak.my-domain.com:1 

Failed to load resource: the server responded with a status of 403 ()

The first 3 messages are actually just warns but the 4th (and last) message was error. My configs:

keycloak docker-compose:

version: '3'
services:
  postgresql:
    image: postgres:16
    environment:
      - POSTGRES_USER=${POSTGRES_USER}
      - POSTGRES_DB=${POSTGRES_DB}
      - POSTGRES_PASSWORD=${POSTGRES_PASSWORD}
    volumes:
      - '/home/guard/docker/keycloak/postgresql_data:/var/lib/postgresql/data'
    networks:
      keycloak:

  keycloak:
    image: quay.io/keycloak/keycloak:26.1
    restart: always
    command: start --hostname https://keycloak.my-domain.com --proxy-headers xforwarded --http-enabled true
    depends_on:
      - postgresql
    environment:
      - KC_PROXY_ADDRESS_FORWARDING=true
      - KC_HTTP_ENABLED=true
      - KC_DB=postgres
      - KC_DB_USERNAME=${POSTGRES_USER}
      - KC_DB_PASSWORD=${POSTGRES_PASSWORD}
      - KC_DB_URL_HOST=postgresql
      - KC_DB_URL_PORT=5432
      - KC_DB_URL_DATABASE=keycloak
      - KEYCLOAK_ADMIN=${KEYCLOAK_ADMIN}
      - KEYCLOAK_ADMIN_PASSWORD=${KEYCLOAK_ADMIN_PASSWORD}

    networks:
      proxy:
      keycloak:

networks:
  proxy:
    external: true
  keycloak:

Caddyfile (keycloak machine):

{
  debug
  email email@gmail.com
}

*.internal.dashrave.eu {
  tls {
    dns cloudflare {env.CLOUDFLARE_API_TOKEN}
  }

  @keycloak host keycloak.internal.my-domain.com
  handle @keycloak {
    reverse_proxy keycloak-keycloak-1:8080
  }
}

Caddyfile (main VM):

{
  debug
  email email@gmail.com
}

keycloak.my-domain.com {
  reverse_proxy https://keycloak.internal.my-domain.com {
    header_up Host {upstream_hostport}
    header_up X-Forwarded-For {remote}
    header_up X-Forwarded-Proto {scheme}
  }
}

Is this still Caddy related? Because at this point i am not sure where i should ask for help.

Since 403 occurs when a server understands the request but refuses to authorize it, (due to insufficient permissions, incorrect file permissions, or misconfigured server settings) I recommend you check for IP restrictions first, but ensure your networking is set up to allow external requests.