Caddy + Pterodactyl : JWT console spam

Kraoc · December 11, 2022, 5:22pm

1. Output of `caddy version`:

v2.6.2

2. How I run Caddy:

a. System environment:

Proxmox / VM Debian 11 / Docker+Portainer

b. Command:

Paste command here.

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane. -->

d. My complete Caddy config:

# 2022-12-11

(logs) {
	log {
		level error
	}
}
(debug) {
	debug
	log {
		level debug
	}
}

(redis) {
	redis {
		url 192.168.50.202:6379
	}
}

(olric) {
	olric {
		url olric:3320
	}
}

(souin) {
	allowed_http_verbs GET POST PATCH

	api {
		souin {
			security
		}
	}

	cdn {
		api_key {env.CF_DNS_API_TOKEN}
		dynamic true
		email {env.CF_API_EMAIL}
		hostname zogg.fr
		provider cloudflare
		strategy soft
	}

	headers Content-Type Authorization

	#log_level debug
	log_level error

	#import redis
	import olric

	default_cache_control no-store
}

(cache) {
	order cache before rewrite
	cache {
		import souin
	}
}

(cloudflareTrustedProxies) {
	trusted_proxies 10.0.0.0/8 172.16.0.0/16 192.168.0.0/16 fc00::/7 173.245.48.0/20 103.21.244.0/22 103.22.200.0/22 103.31.4.0/22 141.101.64.0/18 108.162.192.0/18 190.93.240.0/20 188.114.96.0/20 197.234.240.0/22
}
(cloudflare) {
	tls {
		dns cloudflare {env.CF_DNS_API_TOKEN}
		resolvers 1.1.1.1 1.0.0.1
	}

	header {
		Host {upstream_hostport}

		X-Forwarded-Proto {scheme}
		X-Forwarded-For {host}

		defer
	}
}

(reverseProxy) {
	import cloudflareTrustedProxies

	import keepalive

	header_up Cache-Control "public,max-age=86400,s-maxage=86400,max-stale=3600,stale-while-revalidate=86400,stale-if-error=86400"
	header_down Cache-Control "public,max-age=86400,s-maxage=86400,max-stale=3600,stale-while-revalidate=86400,stale-if-error=86400"

	header_down X-Powered-By "Zogg"

	header_up Host {host}
	header_down Host {host}

	header_up X-Real-IP {host}
	header_down X-Real-IP {host}

	header_up X-Forwarded-For {host}
	header_down X-Forwarded-For {host}

	header_up Content-Security-Policy "default-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob: wss: https:"
	header_down Content-Security-Policy "default-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob: wss: https:"

	header_down -Via
	header_down -X-Varnish
	header_down -X-Url
	header_down -Link
	header_down -X-Host
}

(headersGlobal) {
	X-Powered-By "Zogg"
	
	Host {host}
	X-Real-IP {host}
	X-Forwarded-For {host}
	
	-Server
	-Via
}

(headersSecurity) {
	Referrer-Policy "strict-origin-when-cross-origin"

	Strict-Transport-Security "max-age=31536000;includeSubDomains;preload"
	X-Permitted-Cross-Domain-Policies: "none"

	X-Content-Type-Options "nosniff"

	X-Frame-Options "SAMEORIGIN"

	X-XSS-Protection 0

	Permissions-Policy "fullscreen=(*),display-capture=(self),accelerometer=(),battery=(),camera=(),autoplay=(self),vibrate=(self),geolocation=(self),midi=(self),notifications=(*),push=(*),microphone=(self),magnetometer=(self),gyroscope=(self),payment=(self)"

	Content-Security-Policy "default-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob: wss: https:"
}

(headersRobots) {
	X-Robots-Tag "none,noarchive,nosnippet,notranslate,noimageindex"
}

(headersCaching) {
	Cache-Control "public,max-age=86400,s-maxage=86400,max-stale=3600,stale-while-revalidate=86400,stale-if-error=86400"
}

(keepalive) {
	transport http {
		resolvers 192.168.50.202
		#keepalive_idle_conns 512
		#keepalive_idle_conns_per_host 256
	}
}

(common) {
	encode zstd gzip
	header {
		import headersGlobal
		import headersRobots
		import headersCaching
		import headersSecurity
		defer
	}
}

(pterodadctyl) {
	encode zstd gzip
	header {
		import headersGlobal
		import headersRobots
		Sec-Fetch-Site "cross-site"
		X-Forwarded-Proto "https"
		Access-Control-Allow-Headers "*,Authorization"
		defer
	}
}

{
	import cache
	
	import logs
	#import debug

	admin off

	acme_dns cloudflare {env.CF_DNS_API_TOKEN}
	email {env.CF_API_EMAIL}
}

import /etc/caddy/conf/entries

And the relevant entry:

# 2022-12-09

@panel host panel.zogg.fr
handle @panel {
	import pterodadctyl
	reverse_proxy 192.168.50.203:8001 {
		import reverseProxy
	}
}

3. The problem I’m having:

4. Error messages and/or full log output:

JWT validation error from wings: Error Event [c6c21f0b-70a3-4c62-852d-843eac63f790]: jwt: exp claim is invalid

This is spammed inside console and website show a spinning circle until I clear the browser cache.

5. What I already tried:

Clear browser cache make console log disapear and website showing it’s content.
Until I need to clear again the browser. And again…

6. Links to relevant resources:

francislavoie · December 12, 2022, 4:43am

That’s not a log message from Caddy.

I don’t think this is a Caddy problem. You haven’t given us enough information to go on here. You left many parts of the help topic template empty.

Kraoc · December 12, 2022, 7:21am

You are right

I switched from Traefik to Caddy recently.
Using Traefik I didn’t have this issue.

As my Pterodactyl stack is using the same components, the main change here is Caddy in place of Traefik.

And maybe I did something wrong ?

Pterodactyl is a software to run games with an administrative panel.
There is mainly a daemon (wings) which run games in docker containers and the Panel to give users a good ui.

When I go to a “game view” I get these logs in my console.
I didn’t have them when I was using Traefik. So I’m searching what’s wrong…

francislavoie · December 12, 2022, 8:03am

@matthewpi Do you have any thoughts?

Kraoc · December 12, 2022, 8:18am

As with my previous bugs reports, I can provide logs if needed… Or respond to any question

What I actually think, as I already had this kinds of problems with Traefik, is that the reverse proxy cannot handle throuput of numerous simulaenous requests to the same backend and handling tls encryption.

Maybe I’m wrong… maybe not.
That’s the only point I guess when comparing http to https behaviour.

matthewpi · December 12, 2022, 5:39pm

Is just the Panel proxied or the node as well? I personally run both proxied in my production environments through caddy, without any problems.

Are you still able to reproduce these issues with other webservers as I don’t think a JWT expiry error would occur unless the response is being cached.

An example of the Caddyfile I use can be found here, the only differences are I use HTTPS and a socket file for PHP-FPM. For proxying the nodes, I just use reverse_proxy without any other header manipulation.

francislavoie · December 12, 2022, 5:52pm

Looking at this now, I see an issue:

	@phpRewrite {
		not file favicon.ico
	}
	try_files @phpRewrite {path} {path}/ /index.php?{query}

This isn’t right – try_files does not support matchers, because it itself embeds a matcher. So Caddy is actually doing filesystem checks for a file literally named @phpRewrite on every request, wasting some syscalls.

Also, php_fastcgi has built-in try_files, so I don’t think this actually does anything for you. You can probably remove this entirely and it should work just the same.

	@startsWithDot {
		path \/\.
		not path .well-known
	}
	rewrite @startsWithDot /index.php{uri}

I also think this doesn’t work correctly, because the path matcher does exact matching. You’re not checking for a file that starts with a dot, but rather if it is a request to exactly the path . instead. You need to use a * to do prefix or suffix matching.

And escaping the / doesn’t do anything for you (and might just never match anyway) because this isn’t a regexp matcher. You’d want path_regexp for regexp matching.

Kraoc · December 12, 2022, 6:29pm

both

panel & node on the same ip/vm but with different port.
under a simple docker stack ^^

for french peoples:

Caddy, Docker et Cloudflare
It’s a bit tricky because I use a dirty hack to allow Wings (docker containers run on the fly) to handle/see local path…

Kraoc · December 12, 2022, 6:32pm

I’ll give a try to caddyfile & entries simplifcation bases on @ francislavoie suggestions !

For now (I didn’t test much today… remote work but work anyay) I didn’t saw jwt error.
I also ajusted pterodactyl setup removing volume to localtime and so and setup time to Etc/UTC.

So I keep this thread on hold until some days. But I hope Francis suggestions are right!

Kraoc · December 12, 2022, 6:36pm

Just saw that config simplification let ghost sharing’s cards works now !

Wohooo

Kraoc · December 12, 2022, 6:43pm

Naaaa! Too complex ! (I said this ?)

My stack owns 2 services: panel + wings
(redis & mariadb are on another VM)

My caddyfile owns 2 reverse proxies

@panel host panel.domain.co
handle @panel {
	import pterodadctyl
	reverse_proxy [ip]:[port1] {
		import reverseProxy
	}
}

@node host node.domain.co
handle @node {
	import pterodadctyl
	reverse_proxy [ip]:[port2] {
		import reverseProxy
	}
}

Don’t need to care about nginx/php-fpm…

matthewpi · December 13, 2022, 2:24am

I updated the file to be simpler. I tried my best to convert an existing NGINX config, however a lot of the directives I moved over where not needed. The same link should show the new file.

Thanks for the info!

matthewpi · December 13, 2022, 2:25am

Does the JWT expiry error still occur or is that resolved now?

Kraoc · December 13, 2022, 5:51am

I will probably have no time to check until friday…

But my caddy entries seems like yours except X-XSS-Protection which I set to 0 as reading many articles it’s not really usefull and can lead to security breach. And my Referrer-Policy is set to strict-origin-when-cross-origin (which is the default).

Well, anyway this community is really active and this is great! Happy to be one of
And one day I will be able to help also!

Kraoc · December 15, 2022, 12:19pm

Well I simplified again my Caddyfile by removing many import and “importing” my subdomains includes

I noticed that some import worked when they were in “include” but trigger warning when I use a big Caddyfile with all inside… Strange…

Maybe caddy synthax parser don’t see things in “include” files ?

I tested again yesterday and it went good since monday until I got again this error.
Clearing the cache resolved it but this is strange.

system · January 10, 2023, 5:23pm

This topic was automatically closed after 30 days. New replies are no longer allowed.