1. The problem I’m having:
I have caddy running on a stand alone instance that is load balancing requests to two private machines. I’ve setup health checks on the endpoints. However, when I test the Caddyfile (by running caddy validate --config Caddyfile
) before deploying it I’m getting the following error. The health check returns context canceled
.
2023/06/12 10:57:56.361 INFO http.handlers.reverse_proxy.health_checker.active HTTP request failed {"host": "10.136.133.149:8080", "error": "Get \"http://10.136.133.149:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG events event {"name": "unhealthy", "id": "4f5c3dae-793a-49c3-b6f9-bbb7f6669cfe", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.149:8080"}}
2023/06/12 10:57:56.361 INFO http.handlers.reverse_proxy.health_checker.active HTTP request failed {"host": "10.136.133.148:8080", "error": "Get \"http://10.136.133.148:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG events event {"name": "unhealthy", "id": "1b4a3bb4-4f88-429a-80dd-7bb7e2afb4d3", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.148:8080"}}
However, when I run curl -vL http://10.136.133.149:8080/healthcheck
from the caddy instance I get the following response. The response includes Content-Type: application/json
. However the response is 200
without any errors. The public IP of this endpoint is http://157.245.252.45:8080/healthcheck
* Connected to 10.136.133.149 (10.136.133.149) port 8080 (#0)
> GET /healthcheck HTTP/1.1
> Host: 10.136.133.149:8080
> User-Agent: curl/7.85.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Server: Werkzeug/2.3.6 Python/3.10.7
< Date: Mon, 12 Jun 2023 10:54:44 GMT
< Content-Type: application/json
< Content-Length: 17
< Connection: close
<
{"message":"OK"}
* Closing connection 0
Here is my Caddyfile
{
debug
}
my.hc1node.com:35997 {
reverse_proxy http://10.136.133.148:35997 http://10.136.133.149:35997 {
header_up Host {upstream_hostport}
lb_policy random
health_interval 10s
health_timeout 5s
health_uri /healthcheck
health_port 8080
health_status 2xx
}
}
my.hc1node.com:35998 {
reverse_proxy http://10.136.133.148:35998 http://10.136.133.149:35998 {
lb_policy random
}
}
Does the Caddyfile
require additional parameters to perform a proper health check on this endpoint?
2. Error messages and/or full log output:
2023/06/12 10:57:56.361 INFO http.handlers.reverse_proxy.health_checker.active HTTP request failed {"host": "10.136.133.149:8080", "error": "Get \"http://10.136.133.149:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG events event {"name": "unhealthy", "id": "4f5c3dae-793a-49c3-b6f9-bbb7f6669cfe", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.149:8080"}}
2023/06/12 10:57:56.361 INFO http.handlers.reverse_proxy.health_checker.active HTTP request failed {"host": "10.136.133.148:8080", "error": "Get \"http://10.136.133.148:8080/healthcheck\": context canceled"}
2023/06/12 10:57:56.361 DEBUG events event {"name": "unhealthy", "id": "1b4a3bb4-4f88-429a-80dd-7bb7e2afb4d3", "origin": "http.handlers.reverse_proxy", "data": {"host":"10.136.133.148:8080"}}
3. Caddy version:
v2.6.4 h1:2hwYqiRwk1tf3VruhMpLcYTg+11fCdr8S3jhNAdnPy8=
4. How I installed and ran Caddy:
a. System environment:
Distributor ID: Ubuntu
Description: Ubuntu 22.10
Release: 22.10
Codename: kinetic
DigitalOcean: 2vcpu-2gb-intel,
b. Command:
systemctl start caddy
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateDevices=yes
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
d. My complete Caddy config:
{
debug
}
my.hc1node.com:35997 {
reverse_proxy http://10.136.133.148:35997 http://10.136.133.149:35997 {
header_up Host {upstream_hostport}
lb_policy random
health_interval 10s
health_timeout 5s
health_uri /healthcheck
health_port 8080
health_status 2xx
}
}
my.hc1node.com:35998 {
reverse_proxy http://10.136.133.148:35998 http://10.136.133.149:35998 {
lb_policy random
}
}