Healthcheck not working with multiple servers

1. Caddy version (caddy version):

Caddy 2.4.5

2. How I run Caddy:

start caddy with json config

a. System environment:

Plain linux

b. Command:

Paste command here.

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.

d. My complete Caddyfile or JSON config:

{
    "logging": {
        "logs": {
            "default": {
                "level": "debug"
            }
        }
    },
    "admin": {
        "disabled": false,
        "listen": "0.0.0.0:2019"
    },
    "apps": {
        "http": {
            "servers": {
                "example": {
                    "listen": [
                        ":2015"
                    ],
                    "experimental_http3": true,
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "reverse_proxy",
                                    "transport": {
                                        "protocol": "http",
                                        "tls": {
                                            "server_name": "httpbin.org"
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "httpbin.org:443"
                                        }
                                    ],
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "Host": [
                                                    "{http.reverse_proxy.upstream.hostport}"
                                                ]
                                            }
                                        }
                                    },
                                    "health_checks": {
                                        "active": {
                                            "uri": "/aa",
                                            "expect_status": 200,
                                            "expect_body": "",
                                            "interval": "10s",
                                            "timeout": "5s",
                                            "headers": {
                                                "Host": [
                                                    "httpbin.org"
                                                ]
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                },
                "example2": {
                    "listen": [
                        ":2016"
                    ],
                    "experimental_http3": true,
                    "routes": [
                        {
                            "handle": [
                                {
                                    "handler": "reverse_proxy",
                                    "transport": {
                                        "protocol": "http",
                                        "tls": {
                                            "server_name": "httpbin.org"
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "httpbin.org:443"
                                        }
                                    ],
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "Host": [
                                                    "{http.reverse_proxy.upstream.hostport}"
                                                ]
                                            }
                                        }
                                    },
                                    "health_checks": {
                                        "active": {
                                            "uri": "/",
                                            "expect_status": 200,
                                            "expect_body": "",
                                            "interval": "10s",
                                            "timeout": "5s",
                                            "headers": {
                                                "Host": [
                                                    "httpbin.org"
                                                ]
                                            }
                                        }
                                    }
                                }
                            ]
                        }
                    ]
                }
            }
        }
    }
}

3. The problem I’m having:

In the config you see that i have two servers, one on port 2015, the other 2016. both use the revers-proxy setting for httpbin.org

the first one does a health_check for the path “/aa” which is illegal on httpbin.org and returns a 404. if i only have this server, i get an error (for the healthcheck, which is ok). but as you can see, i have a second server on port 2016 and this has a health_check to “/” and this is ok.

the result is, that both servers are healthy although the first should be unhealthy.

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

I think what’s happening is that since both the upstreams on your two separate servers are the same, Caddy only actually has one upstream entry. So health checks for both happen, but since one of them is successful, it constantly overrides the failures and keeps the upstream marked healthy.

You can verify this by making a request to the upstreams API which will show you the internal state of the upstreams list:

I figure that in a real app, you probably wouldn’t use the same dial address for the two servers, so it’s likely not to be an issue.

well, in my config i do use the same dial address for the two servers, because the target is the IP-address of an internal loadbalancer and the “Host” header decides what host i want to balance too. in my concret usecase i have a caddyserver in front of a K8S ingress-controller. caddy has great additional apps so i place it in front. and the ingress i use as the backend is not managed by me.

my only workaround is to patch the /etc/hosts with different entries for the different backends. not nice …

Unfortunately I don’t think it’s possible to separate them right now.

It could be possible to add a new field to the Upstream data structure to add an ID field which would disambiguate the upstreams between servers.

If you’d like to make a feature request, you can open an issue for this on Github.

Issue created here: https://github.com/caddyserver/caddy/issues/4341

2 Likes