Understanding "slow" startup times

1. The problem I’m having:

with a restart of caddy using supervisord (ie. stop and then start), I’ve noticed that it takes quite awhile for caddy to be able to start serving HTTPS, as for a very noticeable period of time, I keep getting “internal error” from the clients, with “no certificate matching TLS ClientHello” in the Caddy logs.

Seems it’s relate to OSCP stapling that first need to happen, so, the question becomes: what is being done before Caddy will server first page succesfully after a restart (no new certificate, using already saved versions)
How could I make that faster?
Or how do I (in a HA/load balancer setup) confirm/check all the certificates is active before handing traffic to the caddy instance?

2. Error messages and/or full log output:

{"level":"debug","ts":1725446554.1754434,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"102.212.62.253","remote_port":"65478","server_name":"ae-tracs-multiplexor01.agileworkscloud.com","remote":"102.212.62.253:65478","identifier":"ae-tracs-multiplexor01.agileworkscloud.com","cipher_suites":[4867,4866,4865,52393,52392,52394,49200,49196,49192,49188,49172,49162,159,107,57,65413,196,136,129,157,61,53,192,132,49199,49195,49191,49187,49171,49161,158,103,51,190,69,156,60,47,186,65,49169,49159,5,4,49170,49160,22,10,255],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}

* Connected to ae-tracs-multiplexor01.agileworkscloud.com (102.212.61.37) port 443
* ALPN: curl offers h2,http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/cert.pem
*  CApath: none
* LibreSSL/3.3.6: error:1404B438:SSL routines:ST_CONNECT:tlsv1 alert internal error
* Closing connection
curl: (35) LibreSSL/3.3.6: error:1404B438:SSL routines:ST_CONNECT:tlsv1 alert internal error

3. Caddy version:

v2.8.4 => github.com/caddyserver/caddy/v2@v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

Observed same with 2.7.4 before I upgrade now to 2.8.4

4. How I installed and ran Caddy:

xcaddy build with these modules:

  Standard modules: 121

dns.providers.acmedns
dns.providers.lego_deprecated
exec
http.handlers.exec
http.handlers.named_route
http.handlers.s3browser
http.handlers.s3proxy
http.handlers.teapot
named_routes

  Non-standard modules: 9

  Unknown modules: 0

using supervisor on Devuan

a. System environment:

Devuan Chimera
LXC inside ProxMox

b. Command:

supervisorctl restart caddy

c. Service/unit/compose file:

d. My complete Caddy config:

{
	log {
		level debug
		format json
		output file /var/log/caddy/global.log {
			roll_size 100M
			roll_keep_for 60d
		}
	}
	default_sni ae-tracs-multiplexor01.agileworkscloud.com
}

ae-tracs-multiplexor01.agileworkscloud.com localhost {
	respond "Hello, world! PLease credit us alot :)<br>

"
}

localhost:2016 {
	respond "Goodbye, world! Hello troubles this far\n"
}

##HH tracsdev02.agileworkscloud.com tracsdev.agileworkscloud.com {
##HH 	reverse_proxy {
##HH 		to https://agw-ae-tracs-mxdev02.qquark.in
##HH 		transport http {
##HH 			tls_insecure_skip_verify
##HH 		}
##HH 		header_up -X-Forwarded-For
##HH 		#         header_up X-Forwarded-Proto {scheme}
##HH 		#         header_up X-Forwarded-Host {host}
##HH 		header_up Host {host}
##HH 	}
##HH }
##h
crush-tracsdev02.agileworkscloud.com {
	reverse_proxy {
		to http://agw-ae-tracs-dev-crusher01.qquark.in:9090
	}
}
##h
##h###
##h
##h
tracs-sam-dev02.agileworkscloud.com {
#	reverse_proxy {
#		to http://agw-ae-tracs-sea-dev01.qquark.in:8888
#		to http://agw-ae-tracs-sea-dev02.qquark.in:8888
#		to http://agw-ae-tracs-sea-dev03.qquark.in:8888
#		to http://agw-ae-tracs-sea-dev04.qquark.in:8888
#		to http://agw-ae-tracs-sea-dev05.qquark.in:8888
#		to http://agw-ae-tracs-sea-dev06.qquark.in:8888
#		health_uri /minio/health/live
#		health_port 9000

#		transport http {
#			versions 1.1
#		}
#		header_up -X-Forwarded-For
	#	header_up X-Forwarded-Proto {scheme}
	#	header_up X-Forwarded-Host {host}
#		header_up Host {host}
#	}
	#        tls {
	#                dns acmedns /etc/acmedns/clientstorage.json
	#                propagation_timeout -1
	#                propagation_delay 5s
	#        }
	log {
		format json
		output file /var/log/caddy/caddy-ae-dev02-sam.log {
		}
		level debug
	}
}
##h
##hagw-ae-dev02-sa.qquark.in {
##h	reverse_proxy {
##h		to http://agw-ae-tracs-sea-dev01.qquark.in:9000
##h		to http://agw-ae-tracs-sea-dev02.qquark.in:9000
##h		to http://agw-ae-tracs-sea-dev03.qquark.in:9000
##h		to http://agw-ae-tracs-sea-dev04.qquark.in:9000
##h		to http://agw-ae-tracs-sea-dev05.qquark.in:9000
##h		to http://agw-ae-tracs-sea-dev06.qquark.in:9000
##h		health_uri /minio/health/live
##h		transport http {
##h			versions 1.1
##h		}
##h		header_up -X-Forwarded-For
##h		header_up X-Forwarded-Proto {scheme}
##h		header_up X-Forwarded-Host {host}
##h		header_up Host {host}
##h	}
##h	tls {
##h		dns acmedns /etc/acmedns/clientstorage.json
##h		propagation_timeout -1
##h		propagation_delay 5s
##h	}
##h	log {
##h		format json
##h		output file /var/log/caddy/caddy-ae-dev02-sa.log {
##h		}
##h		level debug
##h	}
##h}

5. Links to relevant resources:

What’s the full log output (as required in the help template)? There is only a single log line that basically repeats what you said in the section above. So your guess is as good as ours currently.

1 Like

It is much more a theoretical question to understand the workings/reasons/flow of why it could be, as I’m having trouble uploading the ~100MB log file in here, but a link to peruse: https://plik.cloudoffice.co.za/file/jP2QeQtuGaFhXYlW/ys3M7WDYBvjECv1b/STDIN

I’m confused by your config. Is most of it really commented out? Why? It doesn’t seem like most of the plugins you have installed are being used at all in your config. Try using a build of Caddy without those unnecessary plugins.

Your logs seem to suggest like Caddy’s config is being reloaded like every few seconds. That’s kinda crazy. Why? That could certainly mess things up.

1 Like