[solved] Too many path regex in my caddy config to route the endpoints to proper matrix synapse workers causing slow response time

qwqtoday · August 12, 2024, 7:20am

1. The problem I’m having:

I have been starting to use matrix synapse workers recently, and their docs says that I have to route these endpoints to the proper synapse workers using their regex they provided.
There are just too many endpoints there and because I am using too many regex in my caddy config, the caddy response time takes 60ms to do, while directly requesting to my worker only takes 6-8ms.

2. Caddy version:

$ caddy version
v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

3. How I installed and ran Caddy:

I don’t remember how do I install caddy, probably downloading from caddy and putting it to /usr/local/bin, I might have compiled caddy myself.
I am running caddy using systemd.

a. System environment:

Ubuntu 22.04.4 LTS x86_64

b. My complete Caddy config:

https://catgirl.hk {
	reverse_proxy /.well-known/matrix/* https://synapse.catgirl.hk {
		header_up Host {upstream_hostport}
	}
}

https://synapse.catgirl.hk {
	# Well known
	header /.well-known/matrix/* Content-Type application/json
	header /.well-known/matrix/* Access-Control-Allow-Origin *
	respond /.well-known/matrix/server `{"m.server": "synapse.catgirl.hk:443"}`
	respond /.well-known/matrix/client `{"m.homeserver":{"base_url":"https://synapse.catgirl.hk"}}`

	# Sync requests
	@sync expression `path_regexp('^/_matrix/client/(r0|v3)/sync$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3)/events$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3)/initialSync$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3)/rooms/[^/]+/initialSync$')
	`

	# Client API requests
	@client_api expression `path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/createRoom$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/publicRooms$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/joined_members$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/context/.*$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/members$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state$')
	|| path_regexp('^/_matrix/client/v1/rooms/.*/hierarchy$')
	|| path_regexp('^/_matrix/client/(v1|unstable)/rooms/.*/relations/')
	|| path_regexp('^/_matrix/client/v1/rooms/.*/threads$')
	|| path_regexp('^/_matrix/client/unstable/im.nheko.summary/summary/.*$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/account/3pid$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/account/whoami$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/devices$')
	|| path_regexp('^/_matrix/client/versions$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/voip/turnServer$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/event/')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/joined_rooms$')
	|| path_regexp('^/_matrix/client/v1/rooms/.*/timestamp_to_event$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable/.*)/rooms/.*/aliases')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/search$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/user/.*/filter(/|$)')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/directory/room/.*$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/capabilities$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/notifications$')
	`

	# Encryption requests
	@encryption expression `path_regexp('^/_matrix/client/(r0|v3|unstable)/keys/query$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/keys/changes$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/keys/claim$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/room_keys/')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/keys/upload')
	`

	# Registration/login requests
	@registration expression `path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/login$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/register$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/register/available$')
	|| path_regexp('^/_matrix/client/v1/register/m.login.registration_token/validity$')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/password_policy$')
	`
	# Event sending requests
	@event_sending expression `path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/redact')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/send')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/state/')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/(join|invite|leave|ban|unban|kick)$')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/join/')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/knock/')
	|| path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/profile/')
	`
	# Account data requests
	@account_data expression `path_regexp('^/_matrix/client/(r0|v3|unstable)/.*/tags')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/.*/account_data')
	`
	# Receipts requests
	@receipts expression `path_regexp('^/_matrix/client/(r0|v3|unstable)/rooms/.*/receipt')
	|| path_regexp('^/_matrix/client/(r0|v3|unstable)/rooms/.*/read_markers')
	`
	# Presence requests
	@presence expression `path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/presence/')
	`

	# User directory search requests
	@user_directory expression `path_regexp('^/_matrix/client/(r0|v3|unstable)/user_directory/search$')
	`

	@typing expression `path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/rooms/.*/typing')
	`

	@to_device expression `path_regexp('^/_matrix/client/(r0|v3|unstable)/sendToDevice/')
	`

	@push_rules expression `path_regexp('^/_matrix/client/(api/v1|r0|v3|unstable)/pushrules/')
	`

	@media expression `path_regexp('/_matrix/media/')
	|| path_regexp('/_matrix/client/v1/media/')
	|| path_regexp('/_matrix/federation/v1/media/')
	`

	reverse_proxy @media localhost:19901 {
		lb_policy round_robin
	}

	reverse_proxy @push_rules localhost:19301 {
		lb_policy round_robin
	}

	reverse_proxy @typing localhost:19321 {
		lb_policy round_robin
	}
	reverse_proxy @to_device localhost:19301 {
		lb_policy round_robin
	}

	reverse_proxy @sync localhost:19111 localhost:19112 {
		lb_policy round_robin
	}

	reverse_proxy @client_api localhost:19101 localhost:19102 {
		lb_policy round_robin
	}
	reverse_proxy @encryption localhost:19101 localhost:19102 {
		lb_policy round_robin
	}
	reverse_proxy @registration localhost:19101 localhost:19102 {
		lb_policy round_robin
	}
	reverse_proxy @event_sending localhost:19101 localhost:19102 {
		lb_policy round_robin
	}
	reverse_proxy @account_data localhost:19301 {
		lb_policy round_robin
	}
	reverse_proxy @receipts localhost:19301 {
		lb_policy round_robin
	}
	reverse_proxy @presence localhost:19311 {
		lb_policy round_robin
	}

	reverse_proxy @user_directory localhost:19101 localhost:19102 {
		lb_policy round_robin
	}

	# Federation requests
	@federation expression `path_regexp('^/_matrix/federation/v1/event/')
	|| path_regexp('^/_matrix/federation/v1/state/')
	|| path_regexp('^/_matrix/federation/v1/state_ids/')
	|| path_regexp('^/_matrix/federation/v1/backfill/')
	|| path_regexp('^/_matrix/federation/v1/get_missing_events/')
	|| path_regexp('^/_matrix/federation/v1/publicRooms')
	|| path_regexp('^/_matrix/federation/v1/query/')
	|| path_regexp('^/_matrix/federation/v1/make_join/')
	|| path_regexp('^/_matrix/federation/v1/make_leave/')
	|| path_regexp('^/_matrix/federation/(v1|v2)/send_join/')
	|| path_regexp('^/_matrix/federation/(v1|v2)/send_leave/')
	|| path_regexp('^/_matrix/federation/v1/make_knock/')
	|| path_regexp('^/_matrix/federation/v1/send_knock/')
	|| path_regexp('^/_matrix/federation/(v1|v2)/invite/')
	|| path_regexp('^/_matrix/federation/v1/event_auth/')
	|| path_regexp('^/_matrix/federation/v1/timestamp_to_event/')
	|| path_regexp('^/_matrix/federation/v1/exchange_third_party_invite/')
	|| path_regexp('^/_matrix/federation/v1/user/devices/')
	|| path_regexp('^/_matrix/key/v2/query')
	|| path_regexp('^/_matrix/federation/v1/hierarchy/')
	|| path_regexp('^/_matrix/federation/v1/version')
	|| path_regexp('^/_matrix/key/v2/server')
	`

	reverse_proxy @federation localhost:19201 localhost:19202 localhost:19203 localhost:19204 {
		lb_policy round_robin
	}

	reverse_proxy localhost:19601
}

4. Links to relevant resources:

https://element-hq.github.io/synapse/latest/workers.html#synapseappgeneric_worker

5. Question:

How do I improve the response time?
How do I make my config look better?

qwqtoday · August 12, 2024, 7:28am

Solved the performance problem. The problem was because https takes time. But how do I make my config look better?

francislavoie · August 14, 2024, 12:19am

You could use heredocs maybe:

	@federation <<CEL
		path_regexp('^/_matrix/federation/v1/event/')
		|| path_regexp('^/_matrix/federation/v1/state/')
		|| path_regexp('^/_matrix/federation/v1/state_ids/')
		|| path_regexp('^/_matrix/federation/v1/backfill/')
		|| path_regexp('^/_matrix/federation/v1/get_missing_events/')
		|| path_regexp('^/_matrix/federation/v1/publicRooms')
		|| path_regexp('^/_matrix/federation/v1/query/')
		|| path_regexp('^/_matrix/federation/v1/make_join/')
		|| path_regexp('^/_matrix/federation/v1/make_leave/')
		|| path_regexp('^/_matrix/federation/(v1|v2)/send_join/')
		|| path_regexp('^/_matrix/federation/(v1|v2)/send_leave/')
		|| path_regexp('^/_matrix/federation/v1/make_knock/')
		|| path_regexp('^/_matrix/federation/v1/send_knock/')
		|| path_regexp('^/_matrix/federation/(v1|v2)/invite/')
		|| path_regexp('^/_matrix/federation/v1/event_auth/')
		|| path_regexp('^/_matrix/federation/v1/timestamp_to_event/')
		|| path_regexp('^/_matrix/federation/v1/exchange_third_party_invite/')
		|| path_regexp('^/_matrix/federation/v1/user/devices/')
		|| path_regexp('^/_matrix/key/v2/query')
		|| path_regexp('^/_matrix/federation/v1/hierarchy/')
		|| path_regexp('^/_matrix/federation/v1/version')
		|| path_regexp('^/_matrix/key/v2/server')
		CEL

But that seems pretty wild that you’d need to split up routing that heavily. I don’t understand why Synapse designed it this way.

I see some of your reverse_proxy have the same sets of upstreams. You could just merge those matchers together to reduce the amount of reverse_proxy lines.

Also, you probably could just use the default LB policy (random), it performs quite well and avoids needing in-memory state to function. Also saves a couple lines of config per proxy.

system · September 13, 2024, 12:19am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.