Health check failed with HTTP request failed "context canceled" error

[0x3639]

1. The problem I’m having:

I have caddy running on a stand alone instance that is load balancing requests to two private machines. I’ve setup health checks on the endpoints. However, when I test the Caddyfile (by running caddy validate --config Caddyfile) before deploying it I’m getting the following error. The health check returns context canceled.

2023/06/12 10:57:56.361 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "10.136.133.149:8080", "error": "Get \"http://10.136.133.149:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG   events  event   {"name": "unhealthy", "id": "4f5c3dae-793a-49c3-b6f9-bbb7f6669cfe", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.149:8080"}}
2023/06/12 10:57:56.361 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "10.136.133.148:8080", "error": "Get \"http://10.136.133.148:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG   events  event   {"name": "unhealthy", "id": "1b4a3bb4-4f88-429a-80dd-7bb7e2afb4d3", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.148:8080"}}

However, when I run curl -vL http://10.136.133.149:8080/healthcheck from the caddy instance I get the following response. The response includes Content-Type: application/json. However the response is 200 without any errors. The public IP of this endpoint is http://157.245.252.45:8080/healthcheck

* Connected to 10.136.133.149 (10.136.133.149) port 8080 (#0)
> GET /healthcheck HTTP/1.1
> Host: 10.136.133.149:8080
> User-Agent: curl/7.85.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: Werkzeug/2.3.6 Python/3.10.7
< Date: Mon, 12 Jun 2023 10:54:44 GMT
< Content-Type: application/json
< Content-Length: 17
< Connection: close
< 
{"message":"OK"}
* Closing connection 0

Here is my Caddyfile

{
        debug
}

my.hc1node.com:35997 {
        reverse_proxy http://10.136.133.148:35997 http://10.136.133.149:35997 {
                header_up Host {upstream_hostport}
                lb_policy random
                health_interval 10s
                health_timeout 5s
                health_uri /healthcheck
                health_port 8080
                health_status 2xx
        }
}

my.hc1node.com:35998 {
        reverse_proxy http://10.136.133.148:35998 http://10.136.133.149:35998 {
                lb_policy random
        }
}

Does the Caddyfile require additional parameters to perform a proper health check on this endpoint?

2. Error messages and/or full log output:

2023/06/12 10:57:56.361 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "10.136.133.149:8080", "error": "Get \"http://10.136.133.149:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG   events  event   {"name": "unhealthy", "id": "4f5c3dae-793a-49c3-b6f9-bbb7f6669cfe", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.149:8080"}}
2023/06/12 10:57:56.361 INFO    http.handlers.reverse_proxy.health_checker.active       HTTP request failed     {"host": "10.136.133.148:8080", "error": "Get \"http://10.136.133.148:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG   events  event   {"name": "unhealthy", "id": "1b4a3bb4-4f88-429a-80dd-7bb7e2afb4d3", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.148:8080"}}

3. Caddy version:

v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=

4. How I installed and ran Caddy:

a. System environment:

Distributor ID: Ubuntu
Description: Ubuntu 22.10
Release: 22.10
Codename: kinetic
DigitalOcean: 2vcpu-2gb-intel,

b. Command:

systemctl start caddy

c. Service/unit/compose file:

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
        debug
}

my.hc1node.com:35997 {
        reverse_proxy http://10.136.133.148:35997 http://10.136.133.149:35997 {
                header_up Host {upstream_hostport}
                lb_policy random
                health_interval 10s
                health_timeout 5s
                health_uri /healthcheck
                health_port 8080
                health_status 2xx
        }
}

my.hc1node.com:35998 {
        reverse_proxy http://10.136.133.148:35998 http://10.136.133.149:35998 {
                lb_policy random
        }
}

5. Links to relevant resources:

[matt (Matt Holt)]

Thanks for the detailed post.

Does the error occur if you do caddy run as well? Or only caddy validate?

I guess it makes sense if it’s happening on caddy validate since the proxy module is provisioned, which starts the health checker, and then it’s stopped right away as the program exits. So that would probably cancel a context, I’d guess.

[0x3639]

Ha… that’s funny. I never actually tried to run it. I just figured there was an error, so never tried it… Well, it works! Sorry about that and thanks for looking at this “issue”. Love Caddy!

[matt (Matt Holt)]

Glad it works And, never thought about active health checks during caddy validate, so thanks for bringing it to our attention.