On-demand TLS fails for a specific domain only

1. Caddy version:

v2.6.1 h1:EDqo59TyYWhXQnfde93Mmv4FJfYe00dO60zMiEt+pzo=

2. How I installed, and run Caddy:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list

curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-xcaddy-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/xcaddy/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-xcaddy.list

sudo apt update
sudo apt install caddy
sudo apt install xcaddy

sudo apt install golang-go

xcaddy build \
    --with github.com/caddy-dns/glesys

sudo dpkg-divert --divert /usr/bin/caddy.default --rename /usr/bin/caddy
sudo mv ./caddy /usr/bin/caddy.custom
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.default 10
sudo update-alternatives --install /usr/bin/caddy caddy /usr/bin/caddy.custom 50

a. System environment:

Ubuntu 22.04, running using systemd

b. Command:

sudo systemctl start caddy

c. Service/unit/compose file:

Default systemd from the apt install. No docker.

d. My complete Caddy config:

{
	# Enable debug log to catch certificate errors.
	debug

	on_demand_tls {
		ask http://localhost:8000/api/caddy/domain
		interval 2m
		burst 20
	}
}

(dns_challenge) {
	issuer acme {
		email "johan@..."
		propagation_delay "60s"
		propagation_timeout "15m0s"
		dns glesys {
			project "..."
			api_key "..."
		}
	}
}

(server) {
	root * {args.0}

	request_body {
		max_size 40MB
	}

	@notStatic not file
	reverse_proxy @notStatic {args.1} {
		lb_try_duration 30s
		lb_try_interval 1s
	}

	file_server {
		hide index.php
	}

	encode zstd gzip

	# Cache build for 60 days.
	@static {
		file
		path /build/*
	}
	header @static Cache-Control max-age=5184000
}

# Redirect www to naked for the root domain.
www.klubbenonline.se {
	redir https://klubbenonline.se{uri}
}

# Main domains.
klubbenonline.se {
	import server /home/forge/klubbenonline.se/current/public localhost:8000
}

# Subdomains using wildcard certificate with Glesys DNS module.
# https://caddyserver.com/docs/caddyfile/patterns#wildcard-certificates
*.klubbenonline.se, *.preview.klubbenonline.se {
	tls {
		import dns_challenge
	}

	import server /home/forge/klubbenonline.se/current/public localhost:8000
}

staging.klubbenonline.se {
	import server /home/forge/staging.klubbenonline.se/current/public localhost:8001
}

*.staging.klubbenonline.se, *.preview.staging.klubbenonline.se {
	tls {
		import dns_challenge
	}

	import server /home/forge/staging.klubbenonline.se/current/public localhost:8001
}

# Custom domains, uses Caddy on demand certificates.
https:// {
	tls {
		on_demand
	}

	# Redirect www domains to the naked domain.
	@www header_regexp www Host ^www\.(.*)$
	redir @www https://{re.www.1}{uri} 301

	import server /home/forge/klubbenonline.se/current/public localhost:8000
}

3. The problem I’m having:

We have many users that point their own domain to our server. We’re using on-demand tls.

The on-demand certificates for our users domains work on every domain except “sodradalarnasdhk.se”. The subdomain “www.sodradalarnasdhk.se” has a working certificate but we’re redirecting to non-www domains.

The www-subdomain has a Let’s Encrypt certificate, but when checking the logs for “sodradalarnasdhk.se” it’s trying both le’s encrypt and zerossl, and it’s failing. Could it be that they recently renewed a Let’s enctrypt certificate on another service? They moved to us 2023-01-15 but the certificate process has failed each attempt since then.

4. Error messages and/or full log output:

From the start of the log:

sudo journalctl -u caddy -b | grep sodradalarnasdhk
Jan 20 09:31:26: {"level":"debug","ts":1674203486.0681527,"logger":"events","msg":"event","name":"tls_get_certificate","id":"6a564136-e7a0-43c4-9880-9a9c9d8928a8","origin":"tls","data":{"client_hello":{"CipherSuites":[27242,4865,4866,4867,49195,49199,52393,52392,49196,49200,49161,49171,49162,49172,156,157,47,53,10],"ServerName":"sodradalarnasdhk.se","SupportedCurves":[19018,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537,513],"SupportedProtos":null,"SupportedVersions":[2570,772,771,770,769],"Conn":{}}}}
Jan 20 09:31:26: {"level":"debug","ts":1674203486.0685365,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"sodradalarnasdhk.se"}
Jan 20 09:31:26: {"level":"debug","ts":1674203486.068955,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"66.249.70.60","remote_port":"47207","sni":"sodradalarnasdhk.se"}
Jan 20 09:31:26: {"level":"info","ts":1674203486.0766287,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"66.249.70.60","remote_port":"47207","server_name":"sodradalarnasdhk.se"}
Jan 20 09:31:26: {"level":"info","ts":1674203486.0772665,"logger":"tls.obtain","msg":"acquiring lock","identifier":"sodradalarnasdhk.se"}
Jan 20 09:31:26: {"level":"info","ts":1674203486.0789797,"logger":"tls.obtain","msg":"lock acquired","identifier":"sodradalarnasdhk.se"}
Jan 20 09:31:26: {"level":"info","ts":1674203486.0792446,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sodradalarnasdhk.se"}
Jan 20 09:31:26: {"level":"debug","ts":1674203486.0793574,"logger":"events","msg":"event","name":"cert_obtaining","id":"0787fe53-a750-454a-8d2a-b18449344407","origin":"tls","data":{"identifier":"sodradalarnasdhk.se"}}
Jan 20 09:31:26: {"level":"info","ts":1674203486.0804038,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"johan@klubbenonline.se"}
Jan 20 09:31:26: {"level":"info","ts":1674203486.08047,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"johan@klubbenonline.se"}
Jan 20 09:31:26: {"level":"info","ts":1674203486.954476,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jan 20 09:31:26: {"level":"debug","ts":1674203486.9555347,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:31:26: {"level":"debug","ts":1674203486.9555523,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:31:27: {"level":"debug","ts":1674203487.0981634,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:31:28: {"level":"error","ts":1674203488.2370844,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
Jan 20 09:31:28: {"level":"error","ts":1674203488.2371085,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/770581236/159923055297","attempt":1,"max_attempts":3}
Jan 20 09:31:29: {"level":"error","ts":1674203489.402863,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/"}
Jan 20 09:31:29: {"level":"info","ts":1674203489.4035506,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme.zerossl.com/v2/DV90","account":"johan@klubbenonline.se"}
Jan 20 09:31:29: {"level":"info","ts":1674203489.4037483,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme.zerossl.com/v2/DV90","account":"johan@klubbenonline.se"}
Jan 20 09:31:38: {"level":"info","ts":1674203498.7735598,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jan 20 09:31:38: {"level":"debug","ts":1674203498.7739384,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:31:38: {"level":"debug","ts":1674203498.7739506,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:31:40: {"level":"debug","ts":1674203500.9380693,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:31:43: {"level":"error","ts":1674203503.3128915,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
Jan 20 09:31:43: {"level":"error","ts":1674203503.3129404,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/Ap_1FRbjFRWYGUthPCtskw","attempt":1,"max_attempts":3}
Jan 20 09:31:43: {"level":"error","ts":1674203503.3129697,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
Jan 20 09:31:43: {"level":"debug","ts":1674203503.3129978,"logger":"events","msg":"event","name":"cert_failed","id":"df85328e-8bcf-4762-ba3d-fdec42b580a6","origin":"tls","data":{"error":{},"identifier":"sodradalarnasdhk.se","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Jan 20 09:31:43: {"level":"error","ts":1674203503.3130584,"logger":"tls.obtain","msg":"will retry","error":"[sodradalarnasdhk.se] Obtain: [sodradalarnasdhk.se] solving challenge: sodradalarnasdhk.se: [sodradalarnasdhk.se] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":17.234006509,"max_duration":2592000}
Jan 20 09:32:43: {"level":"info","ts":1674203563.3138223,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sodradalarnasdhk.se"}
Jan 20 09:32:43: {"level":"debug","ts":1674203563.3138924,"logger":"events","msg":"event","name":"cert_obtaining","id":"a76f0215-361b-4936-ab26-2130c739a724","origin":"tls","data":{"identifier":"sodradalarnasdhk.se"}}
Jan 20 09:32:44: {"level":"info","ts":1674203564.0950022,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 20 09:32:44: {"level":"debug","ts":1674203564.096138,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:32:44: {"level":"debug","ts":1674203564.0961552,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:32:44: {"level":"debug","ts":1674203564.247046,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:32:45: {"level":"error","ts":1674203565.449571,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
Jan 20 09:32:45: {"level":"error","ts":1674203565.449595,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/73971044/6657453323","attempt":1,"max_attempts":3}
Jan 20 09:32:46: {"level":"info","ts":1674203566.7772088,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 20 09:32:46: {"level":"debug","ts":1674203566.7780302,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:32:46: {"level":"debug","ts":1674203566.7780461,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:32:46: {"level":"debug","ts":1674203566.930195,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:32:47: {"level":"error","ts":1674203567.730108,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/VN6cc1B9D135pV6_vhHB-ZJguwZuDHzyKtBnMUrejic: 404","instance":"","subproblems":[]}}
Jan 20 09:32:47: {"level":"error","ts":1674203567.7301314,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/VN6cc1B9D135pV6_vhHB-ZJguwZuDHzyKtBnMUrejic: 404","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/73971044/6657454023","attempt":2,"max_attempts":3}
Jan 20 09:32:47: {"level":"error","ts":1674203567.7301588,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - 2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/VN6cc1B9D135pV6_vhHB-ZJguwZuDHzyKtBnMUrejic: 404"}
Jan 20 09:32:56: {"level":"info","ts":1674203576.9902058,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jan 20 09:32:56: {"level":"debug","ts":1674203576.9907537,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:32:56: {"level":"debug","ts":1674203576.9908876,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:33:00: {"level":"debug","ts":1674203580.59567,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:33:08: {"level":"error","ts":1674203588.7510295,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
Jan 20 09:33:08: {"level":"error","ts":1674203588.7510595,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/KEuwrH1TNV09Iq2FJVXbJw","attempt":1,"max_attempts":3}
Jan 20 09:33:08: {"level":"error","ts":1674203588.7510881,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
Jan 20 09:33:08: {"level":"debug","ts":1674203588.7511327,"logger":"events","msg":"event","name":"cert_failed","id":"5558694d-d1b9-4a10-b28d-3e2ccfabecb4","origin":"tls","data":{"error":{},"identifier":"sodradalarnasdhk.se","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Jan 20 09:33:08: {"level":"error","ts":1674203588.7511604,"logger":"tls.obtain","msg":"will retry","error":"[sodradalarnasdhk.se] Obtain: [sodradalarnasdhk.se] solving challenge: sodradalarnasdhk.se: [sodradalarnasdhk.se] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":102.672108561,"max_duration":2592000}
Jan 20 09:34:26: {"level":"info","ts":1674203666.0770154,"logger":"tls.obtain","msg":"releasing lock","identifier":"sodradalarnasdhk.se"}
Jan 20 09:45:01: {"level":"debug","ts":1674204301.5197625,"logger":"events","msg":"event","name":"tls_get_certificate","id":"ea704918-e97d-4a65-a9f0-1c27577305dc","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"sodradalarnasdhk.se","SupportedCurves":[29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"Conn":{}}}}
Jan 20 09:45:01: {"level":"debug","ts":1674204301.5198207,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"sodradalarnasdhk.se"}
Jan 20 09:45:01: {"level":"debug","ts":1674204301.5198486,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"212.247.140.230","remote_port":"23399","sni":"sodradalarnasdhk.se"}
Jan 20 09:45:01: {"level":"info","ts":1674204301.5298398,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"212.247.140.230","remote_port":"23399","server_name":"sodradalarnasdhk.se"}
Jan 20 09:45:01: {"level":"info","ts":1674204301.530463,"logger":"tls.obtain","msg":"acquiring lock","identifier":"sodradalarnasdhk.se"}
Jan 20 09:45:01: {"level":"info","ts":1674204301.5322578,"logger":"tls.obtain","msg":"lock acquired","identifier":"sodradalarnasdhk.se"}
Jan 20 09:45:01: {"level":"info","ts":1674204301.532545,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sodradalarnasdhk.se"}
Jan 20 09:45:01: {"level":"debug","ts":1674204301.532676,"logger":"events","msg":"event","name":"cert_obtaining","id":"ee09e217-258c-49a8-ae97-39d7b9e8bdaf","origin":"tls","data":{"identifier":"sodradalarnasdhk.se"}}
Jan 20 09:45:01: {"level":"info","ts":1674204301.5338776,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"johan@klubbenonline.se"}
Jan 20 09:45:01: {"level":"info","ts":1674204301.5339212,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"johan@klubbenonline.se"}
Jan 20 09:45:02: {"level":"error","ts":1674204302.1513884,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/failed-validation-limit/"}
Jan 20 09:45:02: {"level":"info","ts":1674204302.1522553,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme.zerossl.com/v2/DV90","account":"johan@klubbenonline.se"}
Jan 20 09:45:02: {"level":"info","ts":1674204302.1523194,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme.zerossl.com/v2/DV90","account":"johan@klubbenonline.se"}
Jan 20 09:45:10: {"level":"info","ts":1674204310.742326,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jan 20 09:45:10: {"level":"debug","ts":1674204310.7432158,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:45:10: {"level":"debug","ts":1674204310.7432597,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:45:11: {"level":"debug","ts":1674204311.84924,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:45:15: {"level":"error","ts":1674204315.989829,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
Jan 20 09:45:15: {"level":"error","ts":1674204315.989854,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/7S5hclIfDNDDwrvukyxUkQ","attempt":1,"max_attempts":3}
Jan 20 09:45:15: {"level":"error","ts":1674204315.9898808,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
Jan 20 09:45:15: {"level":"debug","ts":1674204315.9899118,"logger":"events","msg":"event","name":"cert_failed","id":"a02f3222-b34f-4ecc-ab7b-6069f0f4fb61","origin":"tls","data":{"error":{},"identifier":"sodradalarnasdhk.se","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Jan 20 09:45:15: {"level":"error","ts":1674204315.9899669,"logger":"tls.obtain","msg":"will retry","error":"[sodradalarnasdhk.se] Obtain: [sodradalarnasdhk.se] solving challenge: sodradalarnasdhk.se: [sodradalarnasdhk.se] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":14.457682705,"max_duration":2592000}
Jan 20 09:46:15: {"level":"info","ts":1674204375.9904091,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sodradalarnasdhk.se"}
Jan 20 09:46:15: {"level":"debug","ts":1674204375.9904995,"logger":"events","msg":"event","name":"cert_obtaining","id":"9f9133ac-2118-4284-aa20-1e624a4c6b1c","origin":"tls","data":{"identifier":"sodradalarnasdhk.se"}}
Jan 20 09:46:16: {"level":"info","ts":1674204376.7678082,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 20 09:46:16: {"level":"debug","ts":1674204376.7689931,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:46:16: {"level":"debug","ts":1674204376.7690096,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:46:16: {"level":"debug","ts":1674204376.920079,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 20 09:46:18: {"level":"error","ts":1674204378.1232047,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
Jan 20 09:46:18: {"level":"error","ts":1674204378.123401,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/73971044/6657679633","attempt":1,"max_attempts":3}
Jan 20 09:46:19: {"level":"info","ts":1674204379.4504943,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 20 09:46:19: {"level":"debug","ts":1674204379.451762,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:46:19: {"level":"debug","ts":1674204379.4517932,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:46:19: {"level":"debug","ts":1674204379.6032867,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:46:20: {"level":"error","ts":1674204380.405556,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/3OlAC8aDPcNevZ1XHpQ-fx5guXV7-WWrN_VhY_G29is: 404","instance":"","subproblems":[]}}
Jan 20 09:46:20: {"level":"error","ts":1674204380.405607,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/3OlAC8aDPcNevZ1XHpQ-fx5guXV7-WWrN_VhY_G29is: 404","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/73971044/6657680543","attempt":2,"max_attempts":3}
Jan 20 09:46:20: {"level":"error","ts":1674204380.4056358,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - 2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/3OlAC8aDPcNevZ1XHpQ-fx5guXV7-WWrN_VhY_G29is: 404"}
Jan 20 09:46:26: {"level":"info","ts":1674204386.7163775,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jan 20 09:46:26: {"level":"debug","ts":1674204386.716805,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:46:26: {"level":"debug","ts":1674204386.7168217,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:46:32: {"level":"debug","ts":1674204392.5847237,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 20 09:46:44: {"level":"error","ts":1674204404.4780388,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
Jan 20 09:46:44: {"level":"error","ts":1674204404.478062,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/TVUE9pGV7SpwE-SBqCXpeA","attempt":1,"max_attempts":3}
Jan 20 09:46:44: {"level":"error","ts":1674204404.4780893,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
Jan 20 09:46:44: {"level":"debug","ts":1674204404.4781172,"logger":"events","msg":"event","name":"cert_failed","id":"f931c99a-3630-48b0-8e10-9776639b5709","origin":"tls","data":{"error":{},"identifier":"sodradalarnasdhk.se","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Jan 20 09:46:44: {"level":"error","ts":1674204404.4781802,"logger":"tls.obtain","msg":"will retry","error":"[sodradalarnasdhk.se] Obtain: [sodradalarnasdhk.se] solving challenge: sodradalarnasdhk.se: [sodradalarnasdhk.se] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":102.945896503,"max_duration":2592000}
Jan 20 09:48:01: {"level":"info","ts":1674204481.5303059,"logger":"tls.obtain","msg":"releasing lock","identifier":"sodradalarnasdhk.se"}

5. What I already tried:

I have tried pointing other new domains to our server and caddy generates the certificate just fine.

6. Links to relevant resources:

It looks like they have recently had other let’s encrypt certs. Only the one for “www.sodradalarnasdhk.se” comes from our server. The user moved to us 2023-01-15.

I would be super grateful for any help or guidance on this issue.

A more recent log file (the post was too long to include it):

sudo journalctl -u caddy -f | grep sodradalarnasdhk
Jan 25 14:12:34: {"level":"debug","ts":1674652354.0100636,"logger":"events","msg":"event","name":"tls_get_certificate","id":"dcf65956-51ec-46b0-a6aa-37785071e312","origin":"tls","data":{"client_hello":{"CipherSuites":[47802,4865,4866,4867,49196,49195,52393,49200,49199,52392,49188,49187,49162,49161,49192,49191,49172,49171,157,156,61,60,53,47,49160,49170,10],"ServerName":"sodradalarnasdhk.se","SupportedCurves":[6682,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[2570,772,771,770,769],"Conn":{}}}}
Jan 25 14:12:34: {"level":"debug","ts":1674652354.0101304,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"sodradalarnasdhk.se"}
Jan 25 14:12:34: {"level":"debug","ts":1674652354.0101619,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"95.194.107.188","remote_port":"54886","sni":"sodradalarnasdhk.se"}
Jan 25 14:12:34: {"level":"info","ts":1674652354.0163023,"logger":"tls.on_demand","msg":"obtaining new certificate","remote_ip":"95.194.107.188","remote_port":"54886","server_name":"sodradalarnasdhk.se"}
Jan 25 14:12:34: {"level":"info","ts":1674652354.016631,"logger":"tls.obtain","msg":"acquiring lock","identifier":"sodradalarnasdhk.se"}
Jan 25 14:12:34: {"level":"info","ts":1674652354.0178936,"logger":"tls.obtain","msg":"lock acquired","identifier":"sodradalarnasdhk.se"}
Jan 25 14:12:34: {"level":"info","ts":1674652354.0180304,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sodradalarnasdhk.se"}
Jan 25 14:12:34: {"level":"debug","ts":1674652354.0180879,"logger":"events","msg":"event","name":"cert_obtaining","id":"89aaa38c-902b-48db-950c-fe7f8fd04478","origin":"tls","data":{"identifier":"sodradalarnasdhk.se"}}
Jan 25 14:12:34: {"level":"info","ts":1674652354.0185342,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"johan@klubbenonline.se"}
Jan 25 14:12:34: {"level":"info","ts":1674652354.018551,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"johan@klubbenonline.se"}
Jan 25 14:12:34: {"level":"info","ts":1674652354.8314183,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jan 25 14:12:34: {"level":"debug","ts":1674652354.8324006,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 25 14:12:34: {"level":"debug","ts":1674652354.832573,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 25 14:12:34: {"level":"debug","ts":1674652354.9900615,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 25 14:12:36: {"level":"error","ts":1674652356.2115448,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
Jan 25 14:12:36: {"level":"error","ts":1674652356.2116857,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/770581236/160922355137","attempt":1,"max_attempts":3}
Jan 25 14:12:37: {"level":"info","ts":1674652357.5679607,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Jan 25 14:12:37: {"level":"debug","ts":1674652357.5684536,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:12:37: {"level":"debug","ts":1674652357.5686371,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:12:37: {"level":"debug","ts":1674652357.7272165,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:12:38: {"level":"error","ts":1674652358.9443407,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/3Nu-P8HXViiMywe_LfAXXhX1FQAscXu0bmiRQgV_Iik: 404","instance":"","subproblems":[]}}
Jan 25 14:12:38: {"level":"error","ts":1674652358.9444487,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/3Nu-P8HXViiMywe_LfAXXhX1FQAscXu0bmiRQgV_Iik: 404","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/770581236/160922359557","attempt":2,"max_attempts":3}
Jan 25 14:12:38: {"level":"error","ts":1674652358.9445612,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - 2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/3Nu-P8HXViiMywe_LfAXXhX1FQAscXu0bmiRQgV_Iik: 404"}
Jan 25 14:12:38: {"level":"info","ts":1674652358.9448855,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme.zerossl.com/v2/DV90","account":"johan@klubbenonline.se"}
Jan 25 14:12:38: {"level":"info","ts":1674652358.945013,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["sodradalarnasdhk.se"],"ca":"https://acme.zerossl.com/v2/DV90","account":"johan@klubbenonline.se"}
Jan 25 14:12:52: {"level":"info","ts":1674652372.8785303,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jan 25 14:12:52: {"level":"debug","ts":1674652372.87914,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:12:52: {"level":"debug","ts":1674652372.879156,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:12:57: {"level":"debug","ts":1674652377.366772,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:13:01: {"level":"error","ts":1674652381.5426908,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
Jan 25 14:13:01: {"level":"error","ts":1674652381.5427134,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/rz0rY-tyAXnLeEXI6B8b_A","attempt":1,"max_attempts":3}
Jan 25 14:13:01: {"level":"error","ts":1674652381.5427346,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
Jan 25 14:13:01: {"level":"debug","ts":1674652381.5427587,"logger":"events","msg":"event","name":"cert_failed","id":"aaa85ae7-9212-4702-b02a-6262ecd08c07","origin":"tls","data":{"error":{},"identifier":"sodradalarnasdhk.se","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Jan 25 14:13:01: {"level":"error","ts":1674652381.542776,"logger":"tls.obtain","msg":"will retry","error":"[sodradalarnasdhk.se] Obtain: [sodradalarnasdhk.se] solving challenge: sodradalarnasdhk.se: [sodradalarnasdhk.se] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":27.524858218,"max_duration":2592000}
Jan 25 14:13:33: {"level":"debug","ts":1674652413.9214056,"logger":"events","msg":"event","name":"tls_get_certificate","id":"064f44a1-b278-4a79-befb-739fea63ba41","origin":"tls","data":{"client_hello":{"CipherSuites":[23130,4865,4866,4867,49196,49195,52393,49200,49199,52392,49188,49187,49162,49161,49192,49191,49172,49171,157,156,61,60,53,47,49160,49170,10],"ServerName":"sodradalarnasdhk.se","SupportedCurves":[60138,29,23,24,25],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,515,2053,2053,1281,2054,1537,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[47802,772,771,770,769],"Conn":{}}}}
Jan 25 14:13:33: {"level":"debug","ts":1674652413.921463,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"sodradalarnasdhk.se"}
Jan 25 14:13:33: {"level":"debug","ts":1674652413.9214785,"logger":"tls.handshake","msg":"all external certificate managers yielded no certificates and no errors","remote_ip":"95.194.107.188","remote_port":"54914","sni":"sodradalarnasdhk.se"}
Jan 25 14:14:01: {"level":"info","ts":1674652441.5429766,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"sodradalarnasdhk.se"}
Jan 25 14:14:01: {"level":"debug","ts":1674652441.5430298,"logger":"events","msg":"event","name":"cert_obtaining","id":"8852f320-6c6c-4bee-9a03-2bf522eca9f7","origin":"tls","data":{"identifier":"sodradalarnasdhk.se"}}
Jan 25 14:14:02: {"level":"info","ts":1674652442.3528354,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 25 14:14:02: {"level":"debug","ts":1674652442.353474,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 25 14:14:02: {"level":"debug","ts":1674652442.353488,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 25 14:14:02: {"level":"debug","ts":1674652442.5114348,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01"}
Jan 25 14:14:03: {"level":"error","ts":1674652443.7383852,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]}}
Jan 25 14:14:03: {"level":"error","ts":1674652443.7384079,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/73971044/6779009233","attempt":1,"max_attempts":3}
Jan 25 14:14:05: {"level":"info","ts":1674652445.0771618,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Jan 25 14:14:05: {"level":"debug","ts":1674652445.0775626,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:14:05: {"level":"debug","ts":1674652445.0775735,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:14:05: {"level":"debug","ts":1674652445.2347953,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:14:06: {"level":"error","ts":1674652446.0739067,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/OarJ53BiHH-7iZaAUH451S_NXGQTcbDZ1y3UWbgeFBU: 404","instance":"","subproblems":[]}}
Jan 25 14:14:06: {"level":"error","ts":1674652446.0739424,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/OarJ53BiHH-7iZaAUH451S_NXGQTcbDZ1y3UWbgeFBU: 404","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/73971044/6779010233","attempt":2,"max_attempts":3}
Jan 25 14:14:06: {"level":"error","ts":1674652446.0739708,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 403 urn:ietf:params:acme:error:unauthorized - 2a00:1968:0:1::18: Invalid response from http://sodradalarnasdhk.se/.well-known/acme-challenge/OarJ53BiHH-7iZaAUH451S_NXGQTcbDZ1y3UWbgeFBU: 404"}
Jan 25 14:14:09: {"level":"info","ts":1674652449.0715714,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Jan 25 14:14:09: {"level":"debug","ts":1674652449.0722065,"logger":"http.acme_client","msg":"waiting for solver before continuing","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:14:09: {"level":"debug","ts":1674652449.0722396,"logger":"http.acme_client","msg":"done waiting for solver","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:14:15: {"level":"debug","ts":1674652455.3528883,"logger":"http.acme_client","msg":"challenge accepted","identifier":"sodradalarnasdhk.se","challenge_type":"http-01"}
Jan 25 14:14:26: {"level":"error","ts":1674652466.799984,"logger":"http.acme_client","msg":"challenge failed","identifier":"sodradalarnasdhk.se","challenge_type":"http-01","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]}}
Jan 25 14:14:26: {"level":"error","ts":1674652466.8000102,"logger":"http.acme_client","msg":"validating authorization","identifier":"sodradalarnasdhk.se","problem":{"type":"","title":"","detail":"","instance":"","subproblems":[]},"order":"https://acme.zerossl.com/v2/DV90/order/96IClqUT2Hrd1LGNKpycNA","attempt":1,"max_attempts":3}
Jan 25 14:14:26: {"level":"error","ts":1674652466.800036,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sodradalarnasdhk.se","issuer":"acme.zerossl.com-v2-DV90","error":"HTTP 0  - "}
Jan 25 14:14:26: {"level":"debug","ts":1674652466.800054,"logger":"events","msg":"event","name":"cert_failed","id":"a4ac4514-fdb1-47e5-ba12-9446d78faeff","origin":"tls","data":{"error":{},"identifier":"sodradalarnasdhk.se","issuers":["acme-v02.api.letsencrypt.org-directory","acme.zerossl.com-v2-DV90"],"renewal":false}}
Jan 25 14:14:26: {"level":"error","ts":1674652466.8000758,"logger":"tls.obtain","msg":"will retry","error":"[sodradalarnasdhk.se] Obtain: [sodradalarnasdhk.se] solving challenge: sodradalarnasdhk.se: [sodradalarnasdhk.se] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":112.782156656,"max_duration":2592000}
Jan 25 14:15:33: {"level":"debug","ts":1674652533.9221716,"logger":"http.stdlib","msg":"http: TLS handshake error from 95.194.107.188:54914: timed out waiting to obtain certificate for sodradalarnasdhk.se"}
Jan 25 14:15:34: {"level":"info","ts":1674652534.0173564,"logger":"tls.obtain","msg":"releasing lock","identifier":"sodradalarnasdhk.se"}

sodradalarnasdhk.se has an AAAA record set.
See:

❯ dig +short A sodradalarnasdhk.se
46.21.106.215
❯ dig +short AAAA sodradalarnasdhk.se
2a00:1968:0:1::18

www.sodradalarnasdhk.se on the other hand:

❯ dig +short A www.sodradalarnasdhk.se
dns.klubbenonline.se.
46.21.106.215
❯ dig +short AAAA www.sodradalarnasdhk.se
dns.klubbenonline.se.

Furthermore, that A and AAAA respond differently:

❯ curl -I sodradalarnasdhk.se -4
HTTP/1.1 308 Permanent Redirect
Connection: close
Location: https://sodradalarnasdhk.se/
Server: Caddy
Date: Wed, 25 Jan 2023 15:19:09 GMT

❯ curl -I sodradalarnasdhk.se -6
HTTP/1.1 302 Found
Server: openresty
Date: Wed, 25 Jan 2023 15:19:11 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 233
Connection: keep-alive
Location: http://idrottonline.se/SodraDalarnasDHK-Draghund/

So I would suggest removing that AAAA record on sodradalarnasdhk.se for now.
But please consider enabling IPv6 on dns.klubbenonline.se :heart:

Also, please update to v2.6.2

1 Like

Thank you - I had completely missed that. The current IPv6 address for sodradalarnasdhk.se is outdated - I will have them fix that. Also we’ll look into enabling IPv6 support for dns.klubbenonline.se.

What would be the best way to upgrade Caddy, given that we made a custom build using xcaddy and later followed this guide Build from source — Caddy Documentation?

Is it safe to update the apt package, then building our new binary, and finally swap binaries using update-alternatives?

You can probably just run sudo caddy upgrade which will ask the Caddy download server to make a build for you with the same plugins but the latest version, or you could just use xcaddy to build again, and replace the custom binary.

Yeah, because you diverted, updating the apt package will not update caddy itself, but only the support files. So you do need to rebuild yourself.

Also, remember that you need to restart the caddy systemd service after upgrading the binary, so that it starts again with the new version. Obviously that will have a short amount of downtime from shutdown/startup.

Thanks.

Does sudo caddy upgrade work even though the module we use, GitHub - caddy-dns/glesys, is not listed here: Download Caddy?

3 posts were split to a new topic: Caddy seemingly binding to only IPv6

It should, yeah. The build server just uses xcaddy under the hood.