1. The problem I’m having:
I’m operating Caddy v2.9.1 on an AWS t4g.small (2 vCPUs, 2 GiB RAM) via systemd. We’re using a custom HTTP endpoint to provide on-demand Custom TLS certificates (tls.get_certificate
). We do not use an ask
endpoint or permission module.Our Requirement is to use our own CA-signed certificates and get the PEM values using HTTP endpoint from the database. Also We want that it should be scalable and on-demand. Once user Point the CNAME to our domain. We can serve the Certificate.
Issue: Caddy is receiving inbound TLS handshakes with SNI equal to its own IP address (or undefined), resulting in Caddy fetching a cert for its IP. These fetches fail with HTTP 403, memory usage grows, and eventually the process is OOM‑killed.
We’re trying to:
- Prevent certificate issuance attempts for non-hostname SNI (e.g., server IP).
- Implement custom SSL via the HTTP endpoint correctly.
- Ensure on-demand SSL works only for authorised, valid hostnames.
2. Error messages and/or full log output:
Jul 10 16:48:24 caddy[552]: {"level":"error","logger":"tls.handshake","msg":"external certificate manager","remote_ip":"[REDACTED]","remote_port":"[REDACTED]","sni":"[REDACTED_IP_OR_EMPTY]","cert_manager":"*caddytls.HTTPCertGetter","cert_manager_idx":0,"error":"got HTTP 403"}
...(repeated many times)...
Jul 10 17:04:31 systemd[1]: caddy.service: A process of this unit has been killed by the OOM killer.
Jul 10 17:04:31 systemd[1]: caddy.service: Consumed 46min 30s CPU, 2 GiB memory peak, 0 B swap.
3. Caddy version:
$ caddy version
v2.9.1
4. How I installed and ran Caddy:
a. System environment:
- Instance: AWS t4g.small (2 vCPU, 2 GiB RAM)
- OS: Linux arm64
- Init: systemd (user
caddy
)
b. Startup command:
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
c. systemd service file:
[Unit]
Description=Caddy web server
After=network.target
[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
[Install]
WantedBy=multi-user.target
d. Caddyfile configuration:
:443 {
reverse_proxy http://[REDACTED_BACKEND]
route /health {
respond 200
}
tls {
on_demand
get_certificate http https://[REDACTED_ISSUER]/ssl-certificate/
}
header {
X-Content-Type-Options nosniff
X-Frame-Options DENY
X-XSS-Protection "1; mode=block"
Referrer-Policy strict-origin
Strict-Transport-Security "max-age=31536000; includeSubDomains"
}
log {
output file /var/log/caddy/access.log {
roll_size 100mb
roll_keep 10
roll_keep_for 720h
}
format json
}
}
:80 {
route /health {
respond 200
}
}
5. Useful references:
tls.get_certificate
directive & on-demand TLS best practices (Caddy Web Server, Caddy Community, Caddy Community)- Known behavior: Caddy will request certs for any SNI, including IP addresses when on-demand TLS is enabled (Caddy Community)
- Documentation recommends using an
ask
endpoint to restrict allowed hostnames (Caddy Community)
Ask:
-
How do I prevent certificate requests when SNI is an IP or not a valid host?
-
Is
403
from issuer expected when the request is invalid? Does it still drain memory due to retry queue or caching? -
What changes should we apply to the Caddyfile:
- Add
ask
endpoint logic? - Restrict SNI based on regex?
- Use
default_sni
oron_demand
only on valid host block?
- Add
-
Any recommended config snippet for custom SSL on-demand using an HTTP endpoint, without causing OOM?
-
Which additional flags, such as
insecure_secrets_log
, or memory profiling tools, should I enable to troubleshoot this?
Thank you! I can share heap profiles, metrics, or expanded logs if needed.