Reverse Proxy with Failover

[aarif (Aarif Rashid)]

1. Caddy version (caddy version):

v2.4.0

2. How I run Caddy:

a. System environment:

macOS

b. Command:

caddy run --watch

3. The problem I’m having:

I am trying to improve a workflow where ngrok is not sufficient.

A third party service I am using sends webhook events and expects a certain response in order to stop it from retrying. In a development environment, I only really care about what the third party sends me when I am debugging. In all other cases, the webhook events can be thrown away. The problem is that without the special text that needs to be sent back and with no client listening to send it back, the service tries to send the event over and over again.

I thought I may be able to solve this with a Caddy/SSH Tunnel solution where the first “upstream” is the local endpoint for an SSH Tunnel and the second would be an endpoint which simply responds with the special text. I tried to simulate this as follows:

# This is simulating the local SSH tunnel
:9000 {
	respond "I am 9000"
}

# This is simulating the special text that needs to be returned
:9001 {
	respond "I am 9001"
}

# This is the reverse proxy endpoint
:8000 {
	reverse_proxy * :9000 :9001 {
		lb_policy first
	}
}

In this configuration, Caddy behaves correctly, responding with “I am 9000” when issuing the following curl command:

curl http://localhost:8000

I then changed the Caddyfile to remove the endpoint at :9000 like so:

# This is simulating the special text that needs to be returned
:9001 {
	respond "I am 9001"
}

# This is the reverse proxy endpoint
:8000 {
	reverse_proxy * :9000 :9001 {
		lb_policy first
	}
}

The expectation was that since the endpoint listening at :9000 was no longer present, I would get the response “I am 9001” when issuing the same curl command. This was not the case.

Can someone suggest what I may be doing wrong or an alternative?

4. Error messages and/or full log output:

2021/05/12 20:06:11.916	ERROR	http.log.error	dial tcp :9000: connect: connection refused	{"request": {"remote_addr": "[::1]:58733", "proto": "HTTP/1.1", "method": "GET", "host": "localhost:8000", "uri": "/", "headers": {"User-Agent": ["curl/7.64.1"], "Accept": ["*/*"]}}, "duration": 0.000221341, "status": 502, "err_id": "42hzktmdi", "err_trace": "reverseproxy.statusError (reverseproxy.go:852)"}

5. What I already tried:

I tried playing with different variations of load balancing durations and intervals along with the health checking parameters to no avail.

6. Links to relevant resources:

[francislavoie]

You need to enable either or both of active or passive health checking. By default, no health checking is performed, so Caddy doesn’t mark the backend as down. You also need to set lb_try_duration to enable retry behaviour if the first backend is not considered available.

Setting fail_duration and lb_try_duration should be enough to get you the behaviour you expect.

[aarif (Aarif Rashid)]

Thank you @francislavoie, I think I’ve figured it out.

This appears to work:

:9001 {
	respond "I am 9001"
}

:8000 {
	reverse_proxy * :9000 :9001 {
		lb_policy first
		lb_try_duration 5s
		lb_try_interval 250ms

		fail_duration 10s
		max_fails 1
		unhealthy_status 5xx
		unhealthy_latency 5s
		unhealthy_request_count 1
	}
}

It appears that the key was making sure fail_duration was greater than lb_try_interval. Which when I thought about it, completely made sense.