Caddy with on demand TLS total outage after an unattended restart when ask endpoint is down

Help
1

1. The problem I’m having:

We run Caddy in front of Apache on a shared hosting server, with on-demand TLS for several thousand customer domains. The ask endpoint is a small PHP script served by the local Apache on 127.0.0.80:80, which answers 200 for a domain we host and 403 otherwise.

Last night Caddy was restarted unattended by a package update at 04:35. At that same moment our ask endpoint was unavailable (our own fault, and unrelated to Caddy — the backend Apache was mid-restart, so the endpoint was refusing connections).

The result was a full TLS outage, and what surprised us is that it was not limited to new domains. In the two minutes following the restart Caddy logged 1187 failed to get permission for on-demand certificate events across 161 distinct hostnames. We checked every one of those 161 against Caddy’s own storage

Those 155 certificates were valid and had been obtained weeks earlier — for example:

2026-08-02 00:57  certificates/acme-v02.api.letsencrypt.org-directory/www.blue-harbour.ch/www.blue-harbour.ch.crt
2026-08-25 12:22  certificates/acme-v02.api.letsencrypt.org-directory/www.northfield-media.ch/www.northfield-media.ch.crt
2026-07-11 07:24  certificates/acme-v02.api.letsencrypt.org-directory/www.cloudmarten.ch/www.cloudmarten.ch.crt
2026-07-29 07:17  certificates/acme-v02.api.letsencrypt.org-directory/www.airport-rides.com/www.airport-rides.com.crt

So after a restart the certificate cache is cold, and it appears every hostname has to pass the ask check again before its certificate can be loaded from storage — even though no new certificate needs to be issued. With ask unreachable, none of them could be served and the sites were down.

Is there a setting that lets Caddy load and serve an existing on-demand certificate from storage when the ask endpoint is temporarily unavailable, and only consult ask when it would actually need to obtain a new certificate?

If there isn’t, we would like to suggest it: the failure mode is quite severe — a brief blip on the ask backend, coinciding with a restart, takes down every site on the server rather than just blocking new issuance. Serving a certificate we already hold does not seem to require a fresh permission check.

We are aware we can (and will) make our ask endpoint more robust. But we would prefer not to have a hard dependency between “a helper endpoint is up at the exact moment Caddy starts” and “certificates we already own can be served”.

Thanks a lot for any guidance.

2. Error messages and/or full log output:

Caddy restart (systemd):

Aug 31 04:35:27 srv01 systemd[1]: Starting Caddy...
Aug 31 04:35:28 srv01 systemd[1]: Started Caddy.

Representative errors immediately after the restart. Domain names have been replaced with fictional ones; everything else is verbatim:

{"level":"error","ts":1788143728.363686,"logger":"tls","msg":"failed to get permission for on-demand certificate","domain":"www.blue-harbour.ch","error":"checking http://127.0.0.80/ask.php to determine if certificate for hostname 'www.blue-harbour.ch' should be allowed: Get \"http://127.0.0.80/ask.php?domain=www.blue-harbour.ch\": dial tcp 127.0.0.80:80: connect: connection refused"}
{"level":"error","ts":1788143728.5098705,"logger":"tls","msg":"failed to get permission for on-demand certificate","domain":"www.northfield-media.ch","error":"checking http://127.0.0.80/ask.php to determine if certificate for hostname 'www.northfield-media.ch' should be allowed: Get \"http://127.0.0.80/ask.php?domain=www.northfield-media.ch\": dial tcp 127.0.0.80:80: connect: connection refused"}
{"level":"error","ts":1788143728.5297127,"logger":"tls","msg":"failed to get permission for on-demand certificate","domain":"www.cloudmarten.ch","error":"checking http://127.0.0.80/ask.php to determine if certificate for hostname 'www.cloudmarten.ch' should be allowed: Get \"http://127.0.0.80/ask.php?domain=www.cloudmarten.ch\": dial tcp 127.0.0.80:80: connect: connection refused"}

Grouped counts for the same window (04:35–04:36):

   1187  tls              failed to get permission for on-demand certificate
    252  http.log.error   dial tcp 127.0.0.80:80: connect: connection refused

Note: our global log level is ERROR, so we do not have debug output for the incident itself.

3. Caddy version:

v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0=

4. How I installed and ran Caddy:

Self-built RPM package, installed with dnf, run under systemd.

a. System environment:

CloudLinux 9.x (RHEL 9 derivative), x86_64, systemd. No Docker. Caddy sits in front of Apache httpd on the same host; Apache listens on 127.0.0.80:80 and serves both the customer sites and the ask endpoint.

b. Command:

systemctl start caddy

c. Service/unit/compose file:

# caddy.service
#
# For using Caddy with a config file.
#
# Make sure the ExecStart and ExecReload commands are correct
# for your installation.
#
# See https://caddyserver.com/docs/install for instructions.
#
# WARNING: This service does not use the --resume flag, so if you
# use the API to make changes, they will be overwritten by the
# Caddyfile next time the service is restarted. If you intend to
# use Caddy's API to configure it, add the --resume flag to the
# `caddy run` command or use the caddy-api.service file instead.

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
StartLimitIntervalSec=0

[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/Caddyfile
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force
TimeoutStopSec=5s
LimitNOFILE=1048576
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_ADMIN CAP_NET_BIND_SERVICE
Restart=always
RestartSec=3
ReadWritePaths=/etc/pki
Environment=GOMEMLIMIT=4096MiB

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	admin 127.0.0.1:8888
	default_bind 127.0.0.1 [::1] 203.0.113.10 [2001:db8::10] 203.0.113.11 [2001:db8::11]
	grace_period 3s

	log {
		output file /var/log/caddy/caddy.log {
			roll_size 250MiB
			roll_keep_for 15d
		}
		level ERROR
	}

	email admin@example-host.net

	on_demand_tls {
		ask http://127.0.0.80/ask.php
	}

	default_sni srv01.example-host.net

	servers {
		protocols h1 h2
		trusted_proxies combine {
			cloudflare {
				interval 12h
				timeout 15s
			}
			static 203.0.113.20 203.0.113.21 203.0.113.22
		}
	}

	import local.d/*.conf
}

# Common options applied to every customer site
(common) {
	reverse_proxy http://127.0.0.80:80 {
		header_up X-Forwarded-Proto {http.request.scheme}
		header_up X-Forwarded-Port {http.request.local.port}
	}
}

http://*.*.*.*,
http://*.*.*.*.*,
http://*.*.*.*.*.* {
	import common
}

# Workaround to match only host names and not ip-addresses:
https://*.*,
https://*.*.*,
https://*.*.*.*,
https://*.*.*.*.*,
https://*.*.*.*.*.* {
	import common
	tls {
		on_demand
		load /etc/caddy/certs
	}
}

5. Links to relevant resources:

Not relevant
2

Unfortunately we won’t be able to help you without the rest of the help template filled out. We can hardly even guess with so little information.

4

Unfortunately we won’t be able to help you without the rest of the help template filled out. We can hardly even guess with so little information.

Hello Matt,

I’ve updated the original post with more informations.

Unfortunately I had to replace domain names and IP addresses from logs and configs with fictional ones or we could be sued for customer information divulgation (RGPD and so on, you know).

Kind regards.

5

We do offer private support at our higher sponsorship tiers for this reason, just so ya know :slight_smile:

Yes, that is working as intended. The ask endpoint must not fail, it should generally be a local, on-device endpoint that is lightweight and reliable.

Our docs could be updated in some places, but that would not have solved/prevented the problem in your case I’m afraid. I will try to get our docs updated to clarify that the endpoint is needed for loading from storage, too, like it already says in our JSON config docs:

A module that will determine whether a certificate is allowed to be loaded from storage or obtained from an issuer on demand.

The reason accessing storage also requires permission is based on legitimate feedback from sponsors who run high-volume sites and spammy storage accesses were very expensive. It makes sense that if a certificate isn’t allowed to be issued, it isn’t allowed to be read from storage either.

6

Yes, and that’s our case. The endpoint is local, but an unexpected event rendered it temporarily unavailable and it raced with an unattended cady restart that ended with a completely dead service. I agree that we should have added a better monitoring on the ask endpoint with alerts though.

I understand this, but I would suggest this could be configurable (default ON to preserve current behavior), because it’s maybe also legitimate to want the existing certificates to be still loaded if there is an issue with the ask endpoint at startup to prevent such an outage.

dial tcp 127.0.0.80:80: connect: connection refused is not a permission refusal though, it’s a connection error.

A refusal would be a returned 403, or anything that is not 2XX actually.

7

You want to configure caddy to start after all other services are reachable.

Consider using Apaches systemd socket mode for this. This starts the listen socket the moment your system starts, programs connecting to it just wait, rather than getting a connection refused.

8

Hi,

Finally I completely replaced the ask http endpoint by a custom caddy module I built that does the permission check from a text file read at startup and then polled frequently for update, containing the authorized domains.

It has some fallback mechanism (including allow the load of the certificate if it already exists in the storage), using a snapshot, if the text file can’t be read or contains garbage, etc.

You can check here if you’re interested or want to comment it:

This should, for us at least, avoid the original issue that made me open this thread :slight_smile:

Kind regards