Internal SSL-error

1. The problem I’m having:

I created a new domain which is supposed to forward everything to another server: proxy0.mapcomplete.org

I restarted caddy, but it couldn’t get a certificate. Trying to connect to the domain gives an internal SSL-error:

2. Error messages and/or full log output:

Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.9088569,"logger":"http","msg":"servers shutting down with eternal grace period"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.909319,"logger":"tls.obtain","msg":"acquiring lock","identifier":"proxy0.mapcomplete.org"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.909684,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.9104006,"logger":"admin.api","msg":"load complete"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.911202,"logger":"tls.obtain","msg":"lock acquired","identifier":"proxy0.mapcomplete.org"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.9113288,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"proxy0.mapcomplete.org"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"debug","ts":1719232522.9113903,"logger":"events","msg":"event","name":"cert_obtaining","id":"38e06a97-56d1-4711-addf-2edc3df736d1","origin":"tls","data":{"identifier":"proxy0.mapcomplete.org"}}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"debug","ts":1719232522.9117339,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme-v02.api.letsencrypt.org-directory"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.9122386,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["proxy0.mapcomplete.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.9122853,"logger":"admin","msg":"stopped previous server","address":"localhost:2019"}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.912306,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["proxy0.mapcomplete.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"info","ts":1719232522.9123235,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1792725067","account_contact":[]}
Jun 24 12:35:22 ubuntu-4gb-hel1-1 systemd[1]: Reloaded Caddy.
Jun 24 12:35:23 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"debug","ts":1719232523.3126297,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; arm64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["746"],"Content-Type":["application/json"],"Date":["Mon, 24 Jun 2024 12:35:23 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jun 24 12:35:23 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"debug","ts":1719232523.3131073,"logger":"http.acme_client","msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1792725067","identifiers":["proxy0.mapcomplete.org"]}
Jun 24 12:35:23 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"debug","ts":1719232523.4444542,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; arm64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 24 Jun 2024 12:35:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["1VRjeNN9xRfzs-TQE5Jn2p2mOLOjqPtfZbcwQqahyCE7JnAR5NU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Jun 24 12:35:23 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"debug","ts":1719232523.589652,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; arm64)"]},"response_headers":{"Boulder-Requester":["1792725067"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["107"],"Content-Type":["application/problem+json"],"Date":["Mon, 24 Jun 2024 12:35:23 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["1VRjeNN94ba0RU4x14LLEEFv7pV2OJ8QVa1xedNULN6fpUggPoc"],"Server":["nginx"]},"status_code":400}
Jun 24 12:35:23 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"error","ts":1719232523.5898173,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"proxy0.mapcomplete.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
Jun 24 12:35:23 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"debug","ts":1719232523.5900552,"logger":"events","msg":"event","name":"cert_failed","id":"104b5187-1971-4cd5-913d-4673d579f224","origin":"tls","data":{"error":{},"identifier":"proxy0.mapcomplete.org","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
Jun 24 12:35:23 ubuntu-4gb-hel1-1 caddy[768400]: {"level":"error","ts":1719232523.5902164,"logger":"tls.obtain","msg":"will retry","error":"[proxy0.mapcomplete.org] Obtain: [proxy0.mapcomplete.org] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.67898804,"max_duration":2592000}

3. Caddy version:

2.8.4

4. How I installed and ran Caddy:

a. System environment:

Ubuntu server 22.04.4 LTS, OpenSSL 3.0.2 15 Mar 2022 (Library: OpenSSL 3.0.2 15 Mar 2022)

b. Command:

systemctl reload caddy

d. My complete Caddy config:

{
	debug
}

hosted.mapcomplete.org {
	root * public/
	file_server
	header {
		+Permissions-Policy "interest-cohort=()"
	}
}

countrycoder.mapcomplete.org {
	root * tiles/
	file_server
	header {
		+Permissions-Policy "interest-cohort=()"
		+Access-Control-Allow-Origin https://hosted.mapcomplete.org https://dev.mapcomplete.org https://mapcomplete.org
	}
}

report.mapcomplete.org {
	reverse_proxy {
		to http://127.0.0.1:2348
	}
}

studio.mapcomplete.org {
	reverse_proxy {
		to http://127.0.0.1:1235
	}
}

lod.mapcomplete.org {
	reverse_proxy /extractgraph {
		to http://127.0.0.1:2346
	}
}

ipinfo.mapcomplete.org {
    reverse_proxy {
        to 127.0.0.1:2347
    }
}

proxy0.mapcomplete.org {
    reverse_proxy {
        to 109.128.57.178:2345
    }
}

The HTTP-01 challenge states “The HTTP-01 challenge can only be done on port 80.”
Best Practice - Keep Port 80 Open

And using the online tool Let’s Debug yields these results https://letsdebug.net/proxy0.mapcomplete.org/2058086

ANotWorking
ERROR
proxy0.mapcomplete.org has an A (IPv4) record (135.181.201.77) but a request to this address over port 80 did not succeed. Your web server must have at least one working IPv4 or IPv6 address.
Get "https://proxy0.mapcomplete.org/.well-known/acme-challenge/letsdebug-test": remote error: tls: internal error

Trace:
@0ms: Making a request to http://proxy0.mapcomplete.org/.well-known/acme-challenge/letsdebug-test (using initial IP 135.181.201.77)
@0ms: Dialing 135.181.201.77
@2ms: Server response: HTTP 308 Permanent Redirect
@2ms: Received redirect to https://proxy0.mapcomplete.org/.well-known/acme-challenge/letsdebug-test
@2ms: Dialing 135.181.201.77
@4ms: Experienced error: remote error: tls: internal error
1 Like

And trying simulating the HTTP-01 Challenge with curl I see

HTTP redirects us to HTTPS

$ curl -Ii http://proxy0.mapcomplete.org/.well-known/acme-challenge/sometestfile
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://proxy0.mapcomplete.org/.well-known/acme-challenge/sometestfile
Server: Caddy
Date: Tue, 25 Jun 2024 02:10:00 GMT

The redirection to HTTPS is failing causing the challenge to fail.

$ curl -k -Ii https://proxy0.mapcomplete.org/.well-known/acme-challenge/sometestfile
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

See if HTTP works on port 80; well a tiny bit; but would cause the challenge to fail.

$ curl -k -Ii http://proxy0.mapcomplete.org:443/.well-known/acme-challenge/sometestfile
HTTP/1.0 400 Bad Request

Both Ports 80 & 443 are Open.

$ nmap -Pn -p80,443 proxy0.mapcomplete.org
Starting Nmap 7.80 ( https://nmap.org ) at 2024-06-25 02:12 UTC
Nmap scan report for proxy0.mapcomplete.org (135.181.201.77)
Host is up (0.18s latency).
rDNS record for 135.181.201.77: static.77.201.181.135.clients.your-server.de

PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.26 seconds

I don’t think that’s true – the redirect only happens if Caddy doesn’t match the challenge to something it can serve, pretty sure. It just falls through to a redirect if the challenge string doesn’t match anything.

We’ve seen JWS verification error before, but I don’t know that we ever figured out why it happens. FYI @matt

One “shotgun” solution is to just wipe out Caddy’s storage then restart Caddy, it should give it a clean slate and it should hopefully work. Might be that something in storage got corrupted somehow, breaking issuance. The storage is at /var/lib/caddy/.local/share/caddy. You could make a backup of it before wiping it, could be useful for debugging, to see what’s in some of those files. I’ll let @matt follow up on that though if he wants to dig deeper.

2 Likes

The shotgun solution of wiping storage worked :slight_smile:

I have a backup of those files, how can I best transfer those files to you?

Note: I have been abusing caddy on this machine (starting it, stopping it at random times, …) so a corruption is very probably

@pietervdvn please take @francislavoie advice, I am only novice with Caddy. :slight_smile:

I do want to figure this out at some point, but for now I am not sure how to troubleshoot it as I’ve never been able to reproduce it organically.

1 Like