error:0A000438:SSL routines::tlsv1 alert internal error

Help
1

1. The problem I’m having:

I have step-ca configured, I generated a certificate with step CLI and added it to Caddy, when I curl Caddy it sometimes fails with error:0A000438:SSL routines::tlsv1 alert internal error (not always, some requests succeed)

Command I used to generated certificate:

> step ca certificate nuc.local.net nuc-local.crt nuc-local.key
✔ Provisioner: REDACTED
Please enter the password to decrypt the provisioner key: 
✔ CA: https://192.168.1.21:8443
✔ Certificate: nuc-local.crt
✔ Private Key: nuc-local.key

When I use same certificates with npx serve (simple web server) everything works with no issues. I wasn’t sure if it’s a Caddy problem or certificate, but since it works for Serve, I assume it’s Caddy

2. Error messages and/or full log output:

There’re no errors in Debug mode of Caddy. The only error I see is on services that are calling Caddy:

> curl https://nuc.local.net/
curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error
> curl https://nuc.local.net/
Hello world!
> curl https://nuc.local.net/
curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Ubuntu 24.04 LTS
OpenSSL 3.0.13 30 Jan 2024 (Library: OpenSSL 3.0.13 30 Jan 2024)

Smallstep CLI/0.26.1 (linux/amd64)
Smallstep CA/0.26.1 (linux/amd64)

b. Command:

sudo caddy run

d. My complete Caddy config:

{
	debug
}

nuc.local.net {
	tls nuc-local.crt nuc-local.key {
		ca https://192.168.1.21:8443
		ca_root /home/user/.step/certs/root_ca.crt
	}

	respond "Hello world!"
}

5. Links to relevant resources:

2

Caddy running as a service usually can’t read files in /home. I suggest finding a way to move that somewhere that Caddy can access it, like /var/lib or something.

Are you sure that’s correct? I think the ACME URL should have a path usually, like /acme/ or something.

It doesn’t quite make sense to me that you’re configuring ca and a cert+key at the same time. If you want to use Step as an ACME issuer, then don’t specify a cert/key, so that Caddy can issue a cert using that issuer, and Caddy will manage that cert in its storage.

I think right now your ca config is being ignored because you specified a cert+key. But you didn’t share your logs, so I have no idea what Caddy is doing. Please share your logs.

3

Thanks for quick review. I don’t want to use ACME for now, I removed ca and ca_root properties, I wasn’t sure if I need them for certificates.

New Caddyfile:

{
        debug
}

nuc.local.net {
        tls nuc-local.crt nuc-local.key {
        }

        respond "Hello world!"
}

Here’re Caddy logs:

2024/05/23 08:18:21.010	INFO	using adjacent Caddyfile
2024/05/23 08:18:21.011	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2024/05/23 08:18:21.011	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc00044b780"}
2024/05/23 08:18:21.011	WARN	tls	stapling OCSP	{"error": "no OCSP stapling for [nuc.local.net]: no OCSP server specified in certificate"}
2024/05/23 08:18:21.011	DEBUG	events	event	{"name": "cached_unmanaged_cert", "id": "290a57a1-5b63-42d8-99fb-e9dfd2df0310", "origin": "tls", "data": {"sans":["nuc.local.net"]}}
2024/05/23 08:18:21.011	DEBUG	tls.cache	added certificate to cache	{"subjects": ["nuc.local.net"], "expiration": "2024/05/24 07:25:07.000", "managed": false, "issuer_key": "", "hash": "c4debc697980f72cc5f721d6f3f7e91338f1bb6f1b30bd01d26ebbdf1764d978", "cache_size": 1, "cache_capacity": 10000}
2024/05/23 08:18:21.011	INFO	http.auto_https	skipping automatic certificate management because one or more matching certificates are already loaded	{"domain": "nuc.local.net", "server_name": "srv0"}
2024/05/23 08:18:21.011	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2024/05/23 08:18:21.011	DEBUG	http.auto_https	adjusted config	{"tls": {"automation":{"policies":[{}]}}, "http": {"servers":{"remaining_auto_https_redirects":{"listen":[":80"],"routes":[{},{}]},"srv0":{"listen":[":443"],"routes":[{"handle":[{"handler":"subroute","routes":[{"handle":[{"body":"Hello world!","handler":"static_response"}]}]}],"terminal":true}],"tls_connection_policies":[{"match":{"sni":["nuc.local.net"]},"certificate_selection":{"any_tag":["cert0"]}},{}],"automatic_https":{}}}}}
2024/05/23 08:18:21.011	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2024/05/23 08:18:21.011	DEBUG	http	starting server loop	{"address": "[::]:443", "tls": true, "http3": true}
2024/05/23 08:18:21.011	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2024/05/23 08:18:21.011	DEBUG	http	starting server loop	{"address": "[::]:80", "tls": false, "http3": false}
2024/05/23 08:18:21.011	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2024/05/23 08:18:21.012	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2024/05/23 08:18:21.012	INFO	serving initial configuration
2024/05/23 08:18:21.012	INFO	tls	cleaning storage unit	{"storage": "FileStorage:/root/.local/share/caddy"}
2024/05/23 08:18:21.013	INFO	tls	finished cleaning storage units
2024/05/23 08:18:44.314	DEBUG	events	event	{"name": "tls_get_certificate", "id": "1738c85d-dbfb-4efe-a2c7-2c0848d2c384", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"nuc.local.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"192.168.1.21","Port":36266,"Zone":""},"LocalAddr":{"IP":"192.168.1.21","Port":443,"Zone":""}}}}
2024/05/23 08:18:44.315	DEBUG	tls.handshake	choosing certificate	{"identifier": "nuc.local.net", "num_choices": 1}
2024/05/23 08:18:44.315	DEBUG	tls.handshake	custom certificate selection results	{"identifier": "nuc.local.net", "subjects": ["nuc.local.net"], "managed": false, "issuer_key": "", "hash": "c4debc697980f72cc5f721d6f3f7e91338f1bb6f1b30bd01d26ebbdf1764d978"}
2024/05/23 08:18:44.315	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "192.168.1.21", "remote_port": "36266", "subjects": ["nuc.local.net"], "managed": false, "expiration": "2024/05/24 07:25:07.000", "hash": "c4debc697980f72cc5f721d6f3f7e91338f1bb6f1b30bd01d26ebbdf1764d978"}
2024/05/23 08:18:45.321	DEBUG	events	event	{"name": "tls_get_certificate", "id": "a30c214b-d81a-4aa4-99c9-039c7e1e8ffd", "origin": "tls", "data": {"client_hello":{"CipherSuites":[4866,4867,4865,49196,49200,159,52393,52392,52394,49195,49199,158,49188,49192,107,49187,49191,103,49162,49172,57,49161,49171,51,157,156,61,60,53,47,255],"ServerName":"nuc.local.net","SupportedCurves":[29,23,30,25,24,256,257,258,259,260],"SupportedPoints":"AAEC","SignatureSchemes":[1027,1283,1539,2055,2056,2057,2058,2059,2052,2053,2054,1025,1281,1537,771,769,770,1026,1282,1538],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"192.168.1.21","Port":36296,"Zone":""},"LocalAddr":{"IP":"192.168.1.21","Port":443,"Zone":""}}}}
2024/05/23 08:18:45.321	DEBUG	tls.handshake	choosing certificate	{"identifier": "nuc.local.net", "num_choices": 1}
2024/05/23 08:18:45.321	DEBUG	tls.handshake	custom certificate selection results	{"identifier": "nuc.local.net", "subjects": ["nuc.local.net"], "managed": false, "issuer_key": "", "hash": "c4debc697980f72cc5f721d6f3f7e91338f1bb6f1b30bd01d26ebbdf1764d978"}
2024/05/23 08:18:45.321	DEBUG	tls.handshake	matched certificate in cache	{"remote_ip": "192.168.1.21", "remote_port": "36296", "subjects": ["nuc.local.net"], "managed": false, "expiration": "2024/05/24 07:25:07.000", "hash": "c4debc697980f72cc5f721d6f3f7e91338f1bb6f1b30bd01d26ebbdf1764d978"}
4

Okay, the logs look fine :thinking: When TLS handshakes fail, typically we’d see a debug log on Caddy’s side with somekind of message, but we’re not seeing that.

Is your .crt file just a single certificate, or is it the chain (leaf + intermediate)?

Would help to see output from curl -v to see why it complains.

5

Exactly, that’s very confusing to me as well.
It’s a chain file.

Nothing really stands out in curl -v

* Host nuc.local.net:443 was resolved.
* IPv6: (none)
* IPv4: 192.168.1.21
*   Trying 192.168.1.21:443...
* Connected to nuc.local.net (192.168.1.21) port 443
* ALPN: curl offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS alert, internal error (592):
* OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error
* Closing connection
curl: (35) OpenSSL/3.0.13: error:0A000438:SSL routines::tlsv1 alert internal error
6

It might be that it was just cert and key file permissions, even though I was running caddy as root, I changed ownership from 600 to 644 and looks like it works now.
It might be a good idea to improve the caddy error message if that was really the case. If error comes back, I’ll send another message here

7

Strange, if Caddy is really running as root (which isn’t necessary btw) the permission change shouldn’t matter.