1. The problem I’m having:
Hi.
- Caddy is behind the LB (HAProxy)
- LB is configured for TCP mode
- Caddy is running inside the Kubernetes
- In our service we give users the opportunity to link their any domain. To set up the domain, they specify the IP of our LB
After starting Caddy, it tries to issue a certificate by specifying IP as the domain name. This is the IP (10.0.0.146) address where Caddy is running. Backend returns an error and Caddy tries again and so on endlessly.
{"level":"debug","time":"2023-11-15T08:54:03.268Z","logger":"tls","msg":"certificate issuance denied","ask_endpoint":"https://…/check-domain","domain":"10.0.0.146","error":"10.0.0.146: certificate not allowed by ask endpoint https://…/check-domain - non-2xx status code 400"}
HTTP works fine. HTTPS doesn’t work, no certificate is issued, it keeps trying to verify 10.0.0.146
2. Error messages and/or full log output:
{"level":"info","ts":1700037313.960676,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":"caddyfile"}
{"level":"info","time":"2023-11-15T08:35:13.964Z","logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","time":"2023-11-15T08:35:13.964Z","logger":"caddy.storage.s3","msg":"use secret_key and access_id for credentials"}
{"level":"warn","time":"2023-11-15T08:35:13.965Z","logger":"http.auto_https","msg":"automatic HTTP->HTTPS redirects are disabled","server_name":"srv1"}
{"level":"debug","time":"2023-11-15T08:35:13.965Z","logger":"http.auto_https","msg":"adjusted config","tls":{"automation":{"policies":[{"on_demand":true}],"on_demand":{"ask":"https://…/check-domain","rate_limit":{"interval":120000000000,"burst":5}}}},"http":{"servers":{"srv0":{"listen":[":3599"],"listener_wrappers":[{"timeout":5000000000,"wrapper":"proxy_protocol"},{"wrapper":"tls"}],"routes":[{"handle":[{"body":"http: Hello World!","handler":"static_response"}]}],"automatic_https":{"disable_redirects":true}},"srv1":{"listen":[":3600"],"listener_wrappers":[{"timeout":5000000000,"wrapper":"proxy_protocol"},{"wrapper":"tls"}],"routes":[{"handle":[{"handler":"reverse_proxy","headers":{"request":{"set":{"X-Real-Ip":["{http.reverse-proxy.upstream.address}"]}},"response":{"set":{"Strict-Transport-Security":["max-age=31536000;"]}}},"transport":{"protocol":"http","tls":{}},"upstreams":[{"dial":"…"}]}]}],"tls_connection_policies":[{}],"automatic_https":{"disable_redirects":true}}}}}
{"level":"info","time":"2023-11-15T08:35:13.965Z","logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002d4000"}
{"level":"info","time":"2023-11-15T08:35:13.965Z","logger":"tls","msg":"cleaning storage unit","description":"S3 Storage Host: …, Bucket: …, Prefix: ssl-data"}
{"level":"info","time":"2023-11-15T08:35:13.965Z","logger":"http","msg":"enabling HTTP/3 listener","addr":":3600"}
{"level":"info","ts":1700037313.9656556,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details."}
{"level":"debug","time":"2023-11-15T08:35:13.965Z","logger":"http","msg":"starting server loop","address":"[::]:3600","tls":true,"http3":true}
{"level":"info","time":"2023-11-15T08:35:13.965Z","logger":"http.log","msg":"server running","name":"srv1","protocols":["h1","h2","h3"]}
{"level":"debug","time":"2023-11-15T08:35:13.965Z","logger":"http","msg":"starting server loop","address":"[::]:3599","tls":false,"http3":false}
{"level":"info","time":"2023-11-15T08:35:13.965Z","logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
{"level":"info","time":"2023-11-15T08:35:13.966Z","msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
{"level":"info","time":"2023-11-15T08:35:13.966Z","msg":"serving initial configuration"}
{"level":"info","time":"2023-11-15T08:35:14.039Z","logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","time":"2023-11-15T08:35:14.354Z","logger":"events","msg":"event","name":"tls_get_certificate","id":"dbeed643-8a43-4012-af3a-1865da8c05e9","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771],"Conn":{"Conn":{}}}}}
{"level":"debug","time":"2023-11-15T08:35:14.354Z","logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.0.0.146"}
{"level":"debug","time":"2023-11-15T08:35:14.827Z","logger":"events","msg":"event","name":"tls_get_certificate","id":"a012f158-6632-4f61-8e0a-2dafd01342a3","origin":"tls","data":{"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":null,"SupportedVersions":[772,771],"Conn":{"Conn":{}}}}}
{"level":"debug","time":"2023-11-15T08:35:14.827Z","logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.0.0.146"}
{"level":"debug","time":"2023-11-15T08:35:15.593Z","logger":"http.stdlib","msg":"http: TLS handshake error from 10.0.0.146:41234: EOF"}
{"level":"debug","time":"2023-11-15T08:35:15.606Z","logger":"tls","msg":"response from ask endpoint","domain":"10.0.0.146","url":"https://…/check-domain?domain=10.0.0.146","status":400}
{"level":"debug","time":"2023-11-15T08:35:15.606Z","logger":"tls","msg":"certificate issuance denied","ask_endpoint":"https://…/check-domain","domain":"10.0.0.146","error":"10.0.0.146: certificate not allowed by ask endpoint https://…/check-domain - non-2xx status code 400"}
{"level":"debug","time":"2023-11-15T08:35:15.606Z","logger":"http.stdlib","msg":"http: TLS handshake error from 10.0.3.145:53072: certificate is not allowed for server name 10.0.0.146: decision func: 10.0.0.146: certificate not allowed by ask endpoint https://…/check-domain - non-2xx status code 400"}
{"level":"debug","time":"2023-11-15T08:35:15.607Z","logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.0.0.146"}
3. Caddy version:
v2.7.5
4. How I installed and ran Caddy:
a. System environment:
Kubernetes / Docker
b. Command:
-
c. Service/unit/compose file:
-
d. My complete Caddy config:
{
debug
log {
level DEBUG
format filter {
wrap json {
time_key time
time_format iso8601
}
}
}
servers {
listener_wrappers {
proxy_protocol {
timeout 5s
}
tls
}
}
auto_https disable_redirects
storage s3 {
…
}
on_demand_tls {
ask https://…/check-domain
interval 2m
burst 5
}
}
:3599 {
respond "http: Hello World!"
}
:3600 {
tls {
on_demand
}
reverse_proxy https://… {
header_up X-Real-IP {http.reverse-proxy.upstream.address}
header_down Strict-Transport-Security max-age=31536000;
}
}
5. Links to relevant resources:
-
I’ll try to set SNI on HAProxy.