Reverse Proxy Backend down

Help
1

Hi all
I wanna know if there is a way to configure Caddy to not trigger the backend timeout after the backend gets down or at least a way to reattach it when it get back online again.
The situation is that I’m proxy a lot of subdomain/backends with Caddy and after one backend get down let’s say that it need to get reloaded for an update Caddy detect it as down and every other request will return a 502 http error and only get fix it by restarting Caddy. There should be a way to don’t mark the backend as failing forever or at least always let all the request pass to the backend without cut it the request in Caddy.
I’m trying to use Caddy in a Continuous Deployment System as a Reverse Proxy for all my backend instances and also for Let’s encript certificate management too.
Here is the structure of my Caddy configuration files

#/etc/caddy/caddy.conf
import /etc/caddy/sites-enabled/*

#/etc/caddy/sites-enabled/demo.caddy
demo.domain.com {
proxy / http://127.0.0.1:9058 {
proxy_header Host {host}
fail_timeout 1800s
}
gzip
log /var/log/caddy/instance-demo_access.log {
rotate {
size 50
age 90 # Keep log files for 90 days
}
}
errors {
log /var/log/caddy/instance-demo_error.log {
size 50
age 90
}
}
}

2

Hey Axel, welcome!

You’ve set your fail_timeout to be 30 minutes. The default is 10 seconds:

fail_timeout specifies how long to consider a backend as down after it has failed. While it is down, requests will not be routed to that backend. A backend is “down” if Caddy fails to communicate with it. The default value is 10 seconds (“10s”).

If you remove that line in your config, a failure will only last for 10 seconds.

1 Like
3

Ok thanks, seems that I misunderstood that line, but that make me wonder if there is a way to disable fail_timeout?, like drive all the request to the backend even if Caddy fails to communicate with it

4

Just out of curiosity what would the purpose of that be? I assume the reason Caddy does this is so you will not lose requests. If caddy can’t communicate with the server, the request will be forwarded then lost.

5

The reason is that I’m loosing requests because caddy has marked the backend as fail and will not forward the request for that backend until the fail timeout has passed. I put fail_timeout = 1 but I’m asking if fail_timeout = 0 will disable it completely and always forward the requests for the backend??

6

Just looked at the code; a fail_timeout of 0 defaults to 10s. Maybe we should reconsider that, and allow fail_timeouts to be disabled. I agree the docs between fail_timeouts and max_fails is confusing… will have to revisit this.

Remember that Caddy can proxy to multiple backends; so once one goes down, it will skip that and use other backends that are still up. If it has to try the down backend on each request (no fail_timeout) then it drastically slows down request handling.

1 Like