Unable to issue a ssl cert via acme dns for netcup

peterge98 · November 27, 2023, 10:17am

1. The problem I’m having:

I am trying to get dns acme working with the netcup api to ensure https on the services hosted in my internal network and that are not published to the internet.

2. Error messages and/or full log output:

root@paperless-ngx:~# ./caddy_linux_amd64_custom run
2023/11/27 10:08:25.701	INFO	using adjacent Caddyfile
2023/11/27 10:08:25.703	WARN	Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies	{"adapter": "caddyfile", "file": "Caddyfile", "line": 11}
2023/11/27 10:08:25.707	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2023/11/27 10:08:25.707	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc000502580"}
2023/11/27 10:08:25.707	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/11/27 10:08:25.707	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/11/27 10:08:25.708	INFO	tls	cleaning storage unit	{"description": "FileStorage:/root/.local/share/caddy"}
2023/11/27 10:08:25.708	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/11/27 10:08:25.708	INFO	tls	finished cleaning storage units
2023/11/27 10:08:25.708	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/27 10:08:25.708	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/27 10:08:25.708	INFO	http	enabling automatic TLS certificate management	{"domains": ["paperless.peterge.de"]}
2023/11/27 10:08:25.708	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2023/11/27 10:08:25.708	INFO	serving initial configuration
2023/11/27 10:08:25.709	INFO	tls.obtain	acquiring lock	{"identifier": "paperless.peterge.de"}
2023/11/27 10:08:25.712	INFO	tls.obtain	lock acquired	{"identifier": "paperless.peterge.de"}
2023/11/27 10:08:25.713	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/27 10:08:25.713	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/27 10:08:25.713	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/27 10:08:26.888	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/11/27 10:08:28.327	ERROR	tls.issuance.acme.acme_client	cleaning up solver	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.paperless.peterge.de\" (usually OK if presenting also failed)"}
2023/11/27 10:08:28.506	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[paperless.peterge.de] solving challenges: presenting for challenge: expected one record, got 0: [] (order=https://acme-v02.api.letsencrypt.org/acme/order/1435642936/225156226566) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2023/11/27 10:08:28.506	INFO	tls.issuance.zerossl	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/27 10:08:28.506	INFO	tls.issuance.zerossl	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/27 10:08:29.447	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/11/27 10:10:33.118	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[paperless.peterge.de] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/1435642936/225156233156) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2023/11/27 10:10:33.118	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/1435642936/225156233156) (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 127.405984022, "max_duration": 2592000}

3. Caddy version:

root@paperless-ngx:~# ./caddy_linux_amd64_custom version
v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

a. System environment:

root@paperless-ngx:~# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 12 (bookworm)"
NAME="Debian GNU/Linux"
VERSION_ID="12"
VERSION="12 (bookworm)"
VERSION_CODENAME=bookworm
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

b. Command:

I downloaded this file:

root@paperless-ngx:~# ls -l caddy_linux_amd64_custom 
-rwxr-xr-x 1 root root 41050112 Nov 27 10:33 caddy_linux_amd64_custom
root@paperless-ngx:~# ./caddy_linux_amd64_custom 
Caddy is an extensible server platform written in Go.
(...)

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

root@paperless-ngx:~# cat Caddyfile 
{
	acme_ca https://acme-v02.api.letsencrypt.org/directory
	email mail@peterge.de
}

paperless.peterge.de {
	tls {
		dns netcup {
			customer_number <5-digit-number>
			api_key <key>
			api_password <pw> 
		}
	}

	reverse_proxy http://localhost:80
}

5. Links to relevant resources:

I am able to see this in netcups api log:

francislavoie · November 27, 2023, 6:29pm

Try disabling the propagation checks in your tls config:

	tls {
		dns netcup {
			customer_number <5-digit-number>
			api_key <key>
			api_password <pw> 
		}
		propagation_timeout -1
	}

peterge98 · November 28, 2023, 10:59am

I can see a lot of request in the API Log:

But it doesnt work with propagation_timeout -1 :

root@paperless-ngx:~# ./caddy_linux_amd64_custom run
2023/11/28 09:31:42.061	INFO	using adjacent Caddyfile
2023/11/28 09:31:42.065	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/11/28 09:31:42.066	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/11/28 09:31:42.066	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/11/28 09:31:42.067	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc000333680"}
2023/11/28 09:31:42.068	INFO	tls	cleaning storage unit	{"description": "FileStorage:/root/.local/share/caddy"}
2023/11/28 09:31:42.069	INFO	tls	finished cleaning storage units
2023/11/28 09:31:42.069	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/28 09:31:42.070	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/11/28 09:31:42.072	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/28 09:31:42.072	INFO	http	enabling automatic TLS certificate management	{"domains": ["paperless.peterge.de"]}
2023/11/28 09:31:42.072	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2023/11/28 09:31:42.072	INFO	serving initial configuration
2023/11/28 09:31:42.074	INFO	tls.obtain	acquiring lock	{"identifier": "paperless.peterge.de"}
2023/11/28 09:31:42.076	INFO	tls.obtain	lock acquired	{"identifier": "paperless.peterge.de"}
2023/11/28 09:31:42.076	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 09:31:42.079	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/28 09:31:42.079	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/28 09:31:43.117	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/11/28 09:31:45.696	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:31:45.696	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/1435642936/225385850966", "attempt": 1, "max_attempts": 3}
2023/11/28 09:31:45.696	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:31:45.697	INFO	tls.issuance.zerossl	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/28 09:31:45.697	INFO	tls.issuance.zerossl	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/28 09:31:46.530	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/11/28 09:31:49.163	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:31:49.163	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-v02.api.letsencrypt.org/acme/order/1435642936/225385858886", "attempt": 1, "max_attempts": 3}
2023/11/28 09:31:49.163	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:31:49.163	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 7.087186126, "max_duration": 2592000}
2023/11/28 09:32:49.165	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 09:32:50.161	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:32:52.769	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:32:52.769	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595214404", "attempt": 1, "max_attempts": 3}
2023/11/28 09:32:52.769	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:32:53.570	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:32:55.974	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:32:55.975	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595215274", "attempt": 1, "max_attempts": 3}
2023/11/28 09:32:55.975	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:32:55.975	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 2, "retrying_in": 120, "elapsed": 73.898674183, "max_duration": 2592000}
2023/11/28 09:34:55.978	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 09:34:56.482	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:34:59.464	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:34:59.464	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595259504", "attempt": 1, "max_attempts": 3}
2023/11/28 09:34:59.464	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:34:59.960	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:35:04.040	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:35:04.041	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595260794", "attempt": 1, "max_attempts": 3}
2023/11/28 09:35:04.041	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:35:04.041	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 3, "retrying_in": 120, "elapsed": 201.964519218, "max_duration": 2592000}
2023/11/28 09:37:04.041	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 09:37:04.542	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:37:07.180	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:37:07.180	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595300074", "attempt": 1, "max_attempts": 3}
2023/11/28 09:37:07.180	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:37:07.672	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:37:11.108	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:37:11.108	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595301604", "attempt": 1, "max_attempts": 3}
2023/11/28 09:37:11.108	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:37:11.108	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 4, "retrying_in": 300, "elapsed": 329.032053373, "max_duration": 2592000}
2023/11/28 09:42:11.111	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 09:42:11.930	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:42:12.613	ERROR	tls.issuance.acme.acme_client	cleaning up solver	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.paperless.peterge.de\" (usually OK if presenting also failed)"}
2023/11/28 09:42:12.772	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[paperless.peterge.de] solving challenges: presenting for challenge: adding temporary record for zone \"peterge.de.\": [netcup] Api session id in invalid format: The session id is not in a valid format. (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595402904) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
2023/11/28 09:42:13.582	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:42:18.156	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:dns", "title": "", "detail": "DNS problem: looking up TXT for _acme-challenge.paperless.peterge.de: DNSSEC: DNSKEY Missing", "instance": "", "subproblems": []}}
2023/11/28 09:42:18.156	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:dns", "title": "", "detail": "DNS problem: looking up TXT for _acme-challenge.paperless.peterge.de: DNSSEC: DNSKEY Missing", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595403464", "attempt": 1, "max_attempts": 3}
2023/11/28 09:42:18.156	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: looking up TXT for _acme-challenge.paperless.peterge.de: DNSSEC: DNSKEY Missing"}
2023/11/28 09:42:18.156	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: looking up TXT for _acme-challenge.paperless.peterge.de: DNSSEC: DNSKEY Missing (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 5, "retrying_in": 600, "elapsed": 636.080337469, "max_duration": 2592000}
2023/11/28 09:52:18.158	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 09:52:19.031	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:52:21.949	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:52:21.949	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595590964", "attempt": 1, "max_attempts": 3}
2023/11/28 09:52:21.949	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:52:22.754	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 09:52:25.759	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 09:52:25.760	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595592184", "attempt": 1, "max_attempts": 3}
2023/11/28 09:52:25.760	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 09:52:25.760	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 6, "retrying_in": 1200, "elapsed": 1243.683543576, "max_duration": 2592000}
2023/11/28 10:12:25.760	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 10:12:26.601	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 10:12:29.964	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 10:12:29.964	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595995134", "attempt": 1, "max_attempts": 3}
2023/11/28 10:12:29.964	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 10:12:30.767	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 10:12:34.157	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 10:12:34.157	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12595996214", "attempt": 1, "max_attempts": 3}
2023/11/28 10:12:34.157	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 10:12:34.157	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ\" (and 3 more) found at _acme-challenge.paperless.peterge.de (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 7, "retrying_in": 1200, "elapsed": 2452.080986404, "max_duration": 2592000}
2023/11/28 10:32:34.158	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/28 10:32:35.063	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 10:32:38.455	ERROR	tls.issuance.acme.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 10:32:38.455	ERROR	tls.issuance.acme.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12596423374", "attempt": 1, "max_attempts": 3}
2023/11/28 10:32:38.455	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 10:32:39.286	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-staging-v02.api.letsencrypt.org/directory"}
2023/11/28 10:32:42.679	ERROR	tls.issuance.zerossl.acme_client	challenge failed	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}}
2023/11/28 10:32:42.679	ERROR	tls.issuance.zerossl.acme_client	validating authorization	{"identifier": "paperless.peterge.de", "problem": {"type": "urn:ietf:params:acme:error:unauthorized", "title": "", "detail": "Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de", "instance": "", "subproblems": []}, "order": "https://acme-staging-v02.api.letsencrypt.org/acme/order/126878214/12596424644", "attempt": 1, "max_attempts": 3}
2023/11/28 10:32:42.680	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de"}
2023/11/28 10:32:42.680	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenge: paperless.peterge.de: [paperless.peterge.de] authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Incorrect TXT record \"ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0\" (and 3 more) found at _acme-challenge.paperless.peterge.de (ca=https://acme-staging-v02.api.letsencrypt.org/directory)", "attempt": 8, "retrying_in": 1800, "elapsed": 3660.60344092, "max_duration": 2592000}
^C2023/11/28 10:58:32.678	INFO	shutting down	{"signal": "SIGINT"}
2023/11/28 10:58:32.678	WARN	exiting; byeee!! 👋	{"signal": "SIGINT"}
2023/11/28 10:58:32.678	INFO	http	servers shutting down with eternal grace period
2023/11/28 10:58:32.681	INFO	tls.obtain	releasing lock	{"identifier": "paperless.peterge.de"}
2023/11/28 10:58:32.681	ERROR	tls.obtain	unable to unlock	{"identifier": "paperless.peterge.de", "lock_key": "issue_cert_paperless.peterge.de", "error": "remove /root/.local/share/caddy/locks/issue_cert_paperless.peterge.de.lock: no such file or directory"}
2023/11/28 10:58:32.681	ERROR	tls	job failed	{"error": "paperless.peterge.de: obtaining certificate: context canceled"}
2023/11/28 10:58:32.681	INFO	admin	stopped previous server	{"address": "localhost:2019"}
2023/11/28 10:58:32.681	INFO	shutdown complete	{"signal": "SIGINT", "exit_code": 0}

francislavoie · November 28, 2023, 7:36pm

Looks like Caddy never cleaned up its previous ACME DNS challenges, probably because validation failed.

$ dig txt _acme-challenge.paperless.peterge.de                                                                     

; <<>> DiG 9.18.18-0ubuntu0.22.04.1-Ubuntu <<>> txt _acme-challenge.paperless.peterge.de
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37777
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 3, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;_acme-challenge.paperless.peterge.de. IN TXT

;; ANSWER SECTION:
_acme-challenge.paperless.peterge.de. 3600 IN TXT "ngGftt_V4lT2-MPB7SCR6Z-JtusIYZupRZUgpfztIL0"
_acme-challenge.paperless.peterge.de. 3600 IN TXT "lYONECTVlpMIXJDqawzm0ySR3IxHwhLrw4z2iqB3hEs"
_acme-challenge.paperless.peterge.de. 3600 IN TXT "Ghn804-R7Z7RsCjeEvJcfxGwHsdH9lTHMvB8lc7ZKyI"
_acme-challenge.paperless.peterge.de. 3600 IN TXT "f3xbhIcMN02EEpZGSE38Mpra_eIdn31dtBjQqu0IdTQ"

;; AUTHORITY SECTION:
peterge.de.		3600	IN	NS	root-dns.netcup.net.
peterge.de.		3600	IN	NS	third-dns.netcup.net.
peterge.de.		3600	IN	NS	second-dns.netcup.net.

;; Query time: 120 msec
;; SERVER: 127.0.0.53#53(127.0.0.53) (UDP)
;; WHEN: Tue Nov 28 14:35:01 EST 2023
;; MSG SIZE  rcvd: 371

You should probably remove these by hand from netcup’s web UI and try again, I suppose.

Remove the _acme-challenge.paperless.peterge.de TXT records

peterge98 · November 30, 2023, 3:33pm

I deleted every record starting with _acme manually and tried it again. Its still not working. Any idea why this is happening?

root@paperless-ngx:~# ./caddy_linux_amd64_custom run
2023/11/30 15:28:52.994	INFO	using adjacent Caddyfile
2023/11/30 15:28:52.998	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2023/11/30 15:28:52.998	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/11/30 15:28:52.998	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/11/30 15:28:52.998	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc000415500"}
2023/11/30 15:28:52.999	INFO	tls	cleaning storage unit	{"description": "FileStorage:/root/.local/share/caddy"}
2023/11/30 15:28:52.999	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/11/30 15:28:52.999	INFO	tls	finished cleaning storage units
2023/11/30 15:28:53.002	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/11/30 15:28:53.002	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/11/30 15:28:53.002	INFO	http	enabling automatic TLS certificate management	{"domains": ["paperless.peterge.de"]}
2023/11/30 15:28:53.003	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2023/11/30 15:28:53.003	INFO	serving initial configuration
2023/11/30 15:28:53.004	INFO	tls.obtain	acquiring lock	{"identifier": "paperless.peterge.de"}
2023/11/30 15:28:53.007	INFO	tls.obtain	lock acquired	{"identifier": "paperless.peterge.de"}
2023/11/30 15:28:53.007	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/11/30 15:28:53.010	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/30 15:28:53.011	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/30 15:28:54.055	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/11/30 15:30:56.072	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[paperless.peterge.de] solving challenges: waiting for solver certmagic.solverWrapper to be ready: timed out waiting for record to fully propagate; verify DNS provider configuration is correct - last error: <nil> (order=https://acme-v02.api.letsencrypt.org/acme/order/1435642936/225912064456) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2023/11/30 15:30:56.072	INFO	tls.issuance.zerossl	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/30 15:30:56.072	INFO	tls.issuance.zerossl	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/11/30 15:30:56.999	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
^C2023/11/30 15:31:50.956	INFO	shutting down	{"signal": "SIGINT"}
2023/11/30 15:31:50.956	WARN	exiting; byeee!! 👋	{"signal": "SIGINT"}
2023/11/30 15:31:50.956	INFO	http	servers shutting down with eternal grace period
2023/11/30 15:31:50.959	INFO	admin	stopped previous server	{"address": "localhost:2019"}
2023/11/30 15:31:50.959	INFO	shutdown complete	{"signal": "SIGINT", "exit_code": 0}

francislavoie · December 1, 2023, 2:56am

Caddy is still waiting for propagation checks, meaning Caddy isn’t able to see that your DNS TXT record was updated (i.e. from Caddy’s perspective using your system’s DNS resolver, it’s not seeing the records it tried to set).

That’s why I suggested turning off propagation_timeout, because that check is not strictly necessary for this to work, it’s just a check that Caddy does to try to make sure things worked correctly, but the check can fail if your system’s DNS resolver is misconfigured.

peterge98 · December 2, 2023, 8:01am

Hmm, the server uses my local pihole instance for dns resolving, which uses these upstream dns servers:

I could change them to sth like quad9, but from what I understand how dns works, they will query the netcup dns server when redolving anything ending with peterge.de anyway, so it shouldn’t make a difference?

peterge98 · December 2, 2023, 8:15am

Okay, I changed pihole to use quad9 as upstream and its working without any issue!
Thank you for the tip on dns @francislavoie <3

peterge98 · December 2, 2023, 3:40pm

Wtf, I got this working once. Then I rebooted to see if it does autostart after boot when being started with the start option. I faced a containerd problem and restored the lxc from a backup. Now I can’t get it working anymore. I have changed nothing on the lxc, it is still using pihole (quad9) as dns. I don’t know why this isn’t working anymore. Any idea?
I deleted the acme entries manually…

root@paperless-ngx:~# ./caddy_linux_amd64_custom run
2023/12/02 15:36:30.978	INFO	using adjacent Caddyfile
2023/12/02 15:36:30.982	INFO	admin	admin endpoint started	{"address": "localhost:2019", "enforce_origin": false, "origins": ["//127.0.0.1:2019", "//localhost:2019", "//[::1]:2019"]}
2023/12/02 15:36:30.982	INFO	http.auto_https	server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS	{"server_name": "srv0", "https_port": 443}
2023/12/02 15:36:30.982	INFO	http.auto_https	enabling automatic HTTP->HTTPS redirects	{"server_name": "srv0"}
2023/12/02 15:36:30.983	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc000425780"}
2023/12/02 15:36:30.983	INFO	http	enabling HTTP/3 listener	{"addr": ":443"}
2023/12/02 15:36:30.983	INFO	tls	cleaning storage unit	{"description": "FileStorage:/root/.local/share/caddy"}
2023/12/02 15:36:30.983	INFO	tls	finished cleaning storage units
2023/12/02 15:36:30.984	INFO	http.log	server running	{"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/12/02 15:36:30.987	INFO	http.log	server running	{"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/12/02 15:36:30.987	INFO	http	enabling automatic TLS certificate management	{"domains": ["paperless.peterge.de"]}
2023/12/02 15:36:30.987	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2023/12/02 15:36:30.987	INFO	serving initial configuration
2023/12/02 15:36:30.988	INFO	tls.obtain	acquiring lock	{"identifier": "paperless.peterge.de"}
2023/12/02 15:36:30.990	INFO	tls.obtain	lock acquired	{"identifier": "paperless.peterge.de"}
2023/12/02 15:36:30.990	INFO	tls.obtain	obtaining certificate	{"identifier": "paperless.peterge.de"}
2023/12/02 15:36:30.991	INFO	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/12/02 15:36:30.991	INFO	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/12/02 15:36:31.937	INFO	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/12/02 15:36:31.964	ERROR	tls.issuance.acme.acme_client	cleaning up solver	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.paperless.peterge.de\" (usually OK if presenting also failed)"}
2023/12/02 15:36:32.132	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[paperless.peterge.de] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.paperless.peterge.de\": unexpected response code 'SERVFAIL' for _acme-challenge.paperless.peterge.de. (order=https://acme-v02.api.letsencrypt.org/acme/order/1444350156/226375411756) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2023/12/02 15:36:32.132	INFO	tls.issuance.zerossl	waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/12/02 15:36:32.133	INFO	tls.issuance.zerossl	done waiting on internal rate limiter	{"identifiers": ["paperless.peterge.de"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "mail@peterge.de"}
2023/12/02 15:36:32.915	INFO	tls.issuance.zerossl.acme_client	trying to solve challenge	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/12/02 15:36:32.935	ERROR	tls.issuance.zerossl.acme_client	cleaning up solver	{"identifier": "paperless.peterge.de", "challenge_type": "dns-01", "error": "no memory of presenting a DNS record for \"_acme-challenge.paperless.peterge.de\" (usually OK if presenting also failed)"}
2023/12/02 15:36:33.085	ERROR	tls.obtain	could not get certificate from issuer	{"identifier": "paperless.peterge.de", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[paperless.peterge.de] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.paperless.peterge.de\": unexpected response code 'SERVFAIL' for _acme-challenge.paperless.peterge.de. (order=https://acme-v02.api.letsencrypt.org/acme/order/1444350156/226375413806) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2023/12/02 15:36:33.085	ERROR	tls.obtain	will retry	{"error": "[paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.paperless.peterge.de\": unexpected response code 'SERVFAIL' for _acme-challenge.paperless.peterge.de. (order=https://acme-v02.api.letsencrypt.org/acme/order/1444350156/226375413806) (ca=https://acme-v02.api.letsencrypt.org/directory)", "attempt": 1, "retrying_in": 60, "elapsed": 2.095307871, "max_duration": 2592000}

francislavoie · December 2, 2023, 4:36pm

It’s still a DNS issue, so I’m not sure what to suggest.

peterge98 · December 3, 2023, 10:47am

I was trying this with paperless.peterge.de redirecting to the IP of the server (10.0.4.104) in pihole the whole time, thats why I couldnt resolve _acme-challenge.peterge.de…

I changed the DNS to 9.9.9.9 in /etc/resolv.conf, but now I am getting this error:


|2023/12/03 10:42:54.233|ERROR|tls.issuance.zerossl.acme_client|cleaning up solver|{identifier: paperless.peterge.de, challenge_type: dns-01, error: no memory of presenting a DNS record for \_acme-challenge.paperless.peterge.de\ (usually OK if presenting also failed)}|
|---|---|---|---|---|
|2023/12/03 10:42:54.390|ERROR|tls.obtain|could not get certificate from issuer|{identifier: paperless.peterge.de, issuer: acme-v02.api.letsencrypt.org-directory, error: [paperless.peterge.de] solving challenges: presenting for challenge: could not determine zone for domain \_acme-challenge.paperless.peterge.de\: unexpected response code 'SERVFAIL' for _acme-challenge.paperless.peterge.de. (order=https://acme-v02.api.letsencrypt.org/acme/order/1444837256/226563179996) (ca=https://acme-v02.api.letsencrypt.org/directory)}|
|2023/12/03 10:42:54.390|ERROR|tls.obtain|will retry|{error: [paperless.peterge.de] Obtain: [paperless.peterge.de] solving challenges: presenting for challenge: could not determine zone for domain \_acme-challenge.paperless.peterge.de\: unexpected response code 'SERVFAIL' for _acme-challenge.paperless.peterge.de. (order=https://acme-v02.api.letsencrypt.org/acme/order/1444837256/226563179996) (ca=https://acme-v02.api.letsencrypt.org/directory), attempt: 1, retrying_in: 60, elapsed: 28.698938737, max_duration: 2592000}|

Any idea why this isnt working?

peterge98 · December 4, 2023, 11:25am

Now its randomly working again without doing any changes. I will close this thread.
No idea what was causing these issues, besides the pihole dns thing.

peterge98 · December 7, 2023, 5:08pm

peterge98 · December 9, 2023, 8:58am

See my article for details

system · January 8, 2024, 8:59am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.