How to do forward_auth to a distant authentication server?

1. The problem I’m having:

Hi, I am already using forward_auth to forward auth to my apps (like Radarr or WLED) to an Authentik authentication server that is running on the same machine, using the following configuration:

app.maindomain.com {
    reverse_proxy app:8080
    reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
    forward_auth authentik-server:9000 {
        copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
        uri /outpost.goauthentik.io/auth/caddy
    }
}

This configuration is from Caddy | authentik , and it works perfectly locally.

Now, I have a new server, with a different IP. I am trying to do the same thing with forward_auth, but instead of reaching an Authentik server on the same machine, it is on a remote machine. So I am trying with this configuration:

app.maindomain.com {
    reverse_proxy app:8080
    reverse_proxy /outpost.goauthentik.io/* https://auth.maindomain.com {
        header_up Host {http.reverse_proxy.upstream.hostport}
    } # This part is added according to Authentik documentation as it is https
    forward_auth https://auth.maindomain.com {
        copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
        uri /outpost.goauthentik.io/auth/caddy
    }
}


To sum up, forward_auth on server 1 works but forward_auth on server 2 doesn’t.

But this doesn’t work. To me, it appears to be a Caddy problem and not an Authentik issue. What is happening is that caddy let me access the app directly, without even forwarding the auth to Authentik. There is no auth at all. The distant Authentik server (and the associated caddy reverse proxy) are not receiving any input, which means caddy is not sending anything. It looks like as soon the forward auth address is distant and no longer local, it completely ignores it. There is no error in my caddy log.
The problem could be because Authentik itself is reverse-proxied by Caddy on server 1, and that additional layer may be the problem of forward_auth. But I’m not sure what to do with this.

Any answer, any help is welcome.
Have a great day.

2. Error messages and/or full log output:

My logs are accessible here : Hostux PrivateBin
I am not noticing any particular error.

3. Caddy version:

v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

Caddy Docker Proxy, Docker Compose

a. System environment:

Odroid N2+ 2Gb
ARM64v8

b. Command:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

5. Links to relevant resources:

Is that your entire config? I don’t see a site serving auth.maindomain.com.

Thanks a lot for your answer.

Sorry, I included only the main part. Here is my full config for server 2 :

adguard.maindomain.com {
	reverse_proxy 192.168.1.1:3003
}
homeassistantnantes.maindomain.com {
	reverse_proxy 192.168.1.144:8123
}
kodi.maindomain.com {
	reverse_proxy 192.168.1.144:8080
}
motioneye.maindomain.com {
	reverse_proxy /outpost.goauthentik.io/* https://auth.maindomain.com:443
	forward_auth https://auth.maindomain.com:443 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
musicassistant.maindomain.com {
	reverse_proxy /outpost.goauthentik.io/* https://auth.maindomain.com:443
	forward_auth https://auth.maindomain.com:443 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
openwrt.maindomain.com {
	reverse_proxy 192.168.1.1:80
}
portainer.maindomain.com {
	reverse_proxy portainer:9443 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}
wled.maindomain.com {
	reverse_proxy 192.168.1.100:80
	reverse_proxy /outpost.goauthentik.io/* https://auth.maindomain.com {
		header_up Host {http.reverse_proxy.upstream.host}
	}
	forward_auth https://auth.maindomain.com {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
zigbee.maindomain.com {
	reverse_proxy /outpost.goauthentik.io/* https://auth.maindomain.com {
		header_up Host {http.reverse_proxy.upstream.host}
	}
	forward_auth https://auth.maindomain.com {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}

auth.maindomain.com is the subdomain of my Authentik server, which is on my server 1. Here is the full config for server 1, which includes the reverse proxy for authentik as well as other apps that work with forward_auth because on the same machine:

adguard2.maindomain.com {
	reverse_proxy adguard:3003
}
aide.maindomain.com {
	reverse_proxy mkdocs:8000
}
ask.maindomain.com {
	reverse_proxy ombi:3579
}
auth.maindomain.com {
	reverse_proxy authentik-server:9000
}
cloud.maindomain.com {
	redir /.well-known/caldav /remote.php/dav 301
	redir /.well-known/carddav /remote.php/dav 301
	redir /.well-known/nodeinfo /nextcloud/index.php/.well-known/nodeinfo 301
	redir /.well-known/webfinger /nextcloud/index.php/.well-known/webfinger 301
	reverse_proxy nextcloud:80
}
crafty.maindomain.com {
	reverse_proxy https://crafty:8443 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}
duplicati.maindomain.com {
	reverse_proxy duplicati:8200
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
epicfreegames.maindomain.com {
	reverse_proxy epicfreegames:3000
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
gotify.maindomain.com {
	reverse_proxy gotify:80
}
homeassistant.maindomain.com {
	reverse_proxy homeassistant:8123
}
joal.maindomain.com {
	reverse_proxy joal:3333
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
lidarr.maindomain.com {
	@except not path /api*
	reverse_proxy lidarr:8686
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth @except authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
mail.maindomain.com {
	reverse_proxy mailu-front:80
}
mylar.maindomain.com {
	reverse_proxy mylar:8090
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
onlyoffice.maindomain.com {
	reverse_proxy onlyoffice-documentserver:80
}
openmediavault.maindomain.com {
	reverse_proxy 192.168.1.27:9090
}
portainer.maindomain.com {
	reverse_proxy portainer:9000
}
prowlarr.maindomain.com {
	@except not path /api*
	reverse_proxy prowlarr:9696
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth @except authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
qbittorrent.maindomain.com {
	@except not path /api/v2*
	reverse_proxy qbittorrent:8082
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth @except authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
radarr.maindomain.com {
	@except not path /api*
	reverse_proxy radarr:7878
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth @except authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
read.maindomain.com {
	reverse_proxy komga:8080
}
readarr.maindomain.com {
	@except not path /api*
	reverse_proxy readarr:8787
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth @except authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
maindomain.com {
	reverse_proxy organizr:80
}
sonarr.maindomain.com {
	@except not path /api*
	reverse_proxy sonarr:8989
	reverse_proxy /outpost.goauthentik.io/* authentik-server:9000
	forward_auth @except authentik-server:9000 {
		copy_headers X-Authentik-Username X-Authentik-Groups X-Authentik-Email X-Authentik-Name X-Authentik-Uid X-Authentik-Jwt X-Authentik-Meta-Jwks X-Authentik-Meta-Outpost X-Authentik-Meta-Provider X-Authentik-Meta-App X-Authentik-Meta-Version
		uri /outpost.goauthentik.io/auth/caddy
	}
}
stream.maindomain.com {
	reverse_proxy jellyfin:8096
}
vault.maindomain.com {
	reverse_proxy vaultwarden:80
}
vikunja.maindomain.com {
	reverse_proxy vikunja-frontend:80
	reverse_proxy /api/* vikunja-api:3456
	reverse_proxy /.well-known/* vikunja-api:3456
	reverse_proxy /dav/* vikunja-api:3456
}

I am doing my tests on my WLED subdomain. If it works, I will apply it to other subdomains with forward_auth.

1 Like

You probably need to add trusted_proxies to to your Server 1’s global options, so that it trusts connections coming from Server 2 and correctly retains X-Forwarded-* headers.

Your logs are a mess though. They don’t look like the default formatting that Caddy emits, there’s tons of INF in there that make it extremely hard to follow. I don’t know what you’re using to read the logs, but it’s probably transforming them in a bad way.

You’ll probably want to enable the debug global option (on both servers) to help trace what’s going on between the servers.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.