Sharing active health checks between reverse_proxy blocks

1. The problem I’m having:

I’m currently evaluating the possibility of a switch over to Caddy from a working HAProxy solution for reverse proxying traffic to several backends. One question I have is around sharing active health checks across different reverse_proxy blocks . Here’s the situation.

I want to have a site configuration that’s listening on multiple ports: 443 and 18080. Based on certain criteria such as specific paths, websocket headers etc, I want to reverse proxy using four different reverse proxy blocks. These blocks will actually mostly the same upstream servers. They will be separated so that in specific scenarios, I can independently control max connections, load balancing algorithms etc for each feature. I think that means the active health checks now have to be duplicated for each reverse proxy block. Is there a way of sharing the active health check information like there is in HA Proxy? I don’t mind defining it many times in config, but I’m more concerned about the unnecessary load on the back end servers of duplicate health checks.

2. Error messages and/or full log output:

N/A

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

Downloaded binaries

a. System environment:

Windows Server 2022

b. Command:

caddy run

c. Service/unit/compose file:

N/A

d. My complete Caddy config:

{
	servers :443 {
		name https
		metrics
		log_credentials
	}

	servers {
		metrics
		log_credentials
	}
	debug
}

# HTTPS Configuration
*.myfrontenddomain.com:443, *.myfrontenddomain.com:18080 {
	# Use your own wildcard certificate with relative paths
	tls ./certs/certificate.crt ./certs/privatekey.key

	# Enable logging with DEBUG level
	log {
		output file ./logs/caddy.log {
			roll_size 10mb
			roll_keep 5
			roll_keep_for 720h
		}
		format json
		level DEBUG
	}

	# Conditional checks
	@dashboardpath path /dashboard*
	@ingestionpath expression (path('*/temp/path1/*') || path('*/temp/path2') || path('/api/v1/temp/logs/audit/*'))
	@websockets {
		header Connection *Upgrade*
		header Upgrade websocket
	}
	@applystickauthheader expression (path('*/temp/path1/*') || path('*/temp/path2') || path('/api/v1/temp/logs/audit/*')) && ({header.Authorization}!='')
	@applystickcookie expression (path('*/temp/path1/*') || path('*/temp/path2') || path('/api/v1/temp/logs/audit/*')) && header_regexp('MYSESSION', 'Cookie', '(.*; )?MYSESSION=([^;]*)(;.*)?')

	# Set X-Sticky header for matched paths and conditions
	request_header @applystickcookie X-Sticky "{cookie.MYSESSION}"
	request_header @applystickauthheader X-Sticky "{header.Authorization}"
	
	# Reverse proxy for dashboard
	reverse_proxy @dashboardpath {
		to https://mybackenddomain.com:5006
	}
	
	# Reverse proxy for websockets
	reverse_proxy @websockets {
		to https://mybackenddomain.com:5006 https://mybackenddomain.com:5007 https://mybackenddomain.com:5008 https://mybackenddomain.com:5009 https://mybackenddomain.com:5010

		# Load Balancing
		lb_policy least_conn

		# Active Health checks
		health_uri /ping
		health_interval 5s

		# Header manipulation
		header_up Host {upstream_hostport}
	}

	# Reverse proxy with custom sticky session header for ingestion paths
	reverse_proxy @ingestionpath {
		to https://mybackenddomain.com:5006 https://mybackenddomain.com:5007 https://mybackenddomain.com:5008 https://mybackenddomain.com:5009 https://mybackenddomain.com:5010

		# Load Balancing
		lb_policy header X-Sticky {
			fallback least_conn
		}

		# Active Health checks
		health_uri /ping
		health_interval 5s

		# Header manipulation
		header_up Host {upstream_hostport}
	}

	# Reverse proxy for all remaining stateless traffic
	reverse_proxy {
		to https://mybackenddomain.com:5006 https://mybackenddomain.com:5007 https://mybackenddomain.com:5008 https://mybackenddomain.com:5009 https://mybackenddomain.com:5010

		# Load Balancing
		lb_policy least_conn

		# Active Health checks
		health_uri /ping
		health_interval 5s

		# Header manipulation
		header_up Host {upstream_hostport}
	}
}

5. Links to relevant resources:

Sorry for the wait, had a busy week.

So actually, the hosts are stored in a global map (not stored in the config, so it survives reloads etc), so health status is shared across all reverse_proxy handlers. But each reverse_proxy will start their own active health checker. So that means your server will be bursting 4 health checks per upstream at the same time, as-is, I think.

FYI, access logs only ever log at INFO and ERROR level, so this won’t do anything for you.

You can simplify this expression matcher:

@ingestionpath `path('*/temp/path1/*', '*/temp/path2', '/api/v1/temp/logs/audit/*')`

Or, in this case you don’t need the expression matcher at all and you can use the path matcher directly:

@ingestionpath path */temp/path1/* */temp/path2 /api/v1/temp/logs/audit/*

You could simplify your upstream list to just mybackenddomain.com:5006-5010, but you’ll need to add the tls option to make it use TLS (whereas https:// would normally turn that on). See reverse_proxy (Caddyfile directive) — Caddy Documentation

1 Like

Thanks for your response. So it sounds like sharing of health check is potentially possible, but I’m still not clear on how to achieve that without multiple hits to the same health check endpoint. I need to ensure that servers are removed from the LB pool for all my reverse proxies if the health check fails.

They would be.

There’s just nothing to stop multiple active health check routines from running concurrently for the same upstream if you set it up in multiple handlers.

The tricky part is each handler could have different config, and someone might have a usecase for having two different handlers with similar upstream lists with different health check config, so I don’t think we can safely deduplicate that without potentially breaking something for someone.

It’s a valid point that there’s advantages of HAProxy’s approach to defining upstreams. That would require a pretty significant redesign of reverse_proxy to allow that sort of thing though, so we’re not planning on taking that approach anytime soon.

2 Likes

Thanks for clarifying. There’s always going to be pros and cons to each solution, so that’s understandable.

So if I load balance on a reverse proxy and add an active health check, then I define another reverse proxy with load balancer that has the same backend servers for the load balancer but no active health check defined, will the health check not apply to the second reverse proxy I.e. unhealthy servers won’t be removed from the pool of the second reverse proxy load balancer?

Also previously you mentioned about simplifying my path expressions. Was that just for readability or do you expect performance differences? Reason I ask is because I’m seeing quite a bit higher CPU under load test when compared to HA proxy. It’s https on the front and back end so I expect encryption overhead etc, but was wondering if the inefficiency of matchers might contribute. I’ve not taken up your suggested improvement yet…

The health status is shared, so they would still get marked unhealthy.

You can test it pretty easily locally by having Caddy proxy & load-balance to itself (add extra site blocks like :8001 and :8002 or w/e and proxy to those, reload the config with them turned on/off with load balancer pointing to them and see what happens.

Readability mainly, but it should be very slightly more memory efficient to have a single path matcher instance than multiple. Also a path matcher not in an expression should also be slightly more efficient because it doesn’t need to call a compiled expression.

Unlikely to be the reason.

2 Likes