After a long wait (almost 12hrs!!) it looks like I managed to get a staging cert for just homebox.
, so now I’m uncommenting the other endpoints and retrying. Going to be a long wait as those are also facing the same issue.
Meanwhile…is it possible to increase the timeout period (and if so, what’s the default timeout)?
EDIT: Further evidence to support that theory as I just saw paperless
succeed after a couple failures. See below filtered log - timestamp difference being noteworthy:
Mar 03 13:18:03 revprox caddy[14584]: {"level":"info","ts":1709471883.2014885,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:18:03 revprox caddy[14584]: {"level":"debug","ts":1709471883.2020526,"logger":"events","msg":"event","name":"cert_obtaining","id":"87b8017b-6144-491f-bf81-6cfbda2cfecf","origin":"tls","data":{"identifier":"paperless.zi0x.duckdns.org"}}
Mar 03 13:18:03 revprox caddy[14584]: {"level":"info","ts":1709471883.8790483,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:18:04 revprox caddy[14584]: {"level":"debug","ts":1709471884.1303015,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"debug","ts":1709471917.3850908,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"error","ts":1709471917.8557284,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"paperless.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.222.19.97:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988335173) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"debug","ts":1709471917.8558824,"logger":"events","msg":"event","name":"cert_failed","id":"b4ca1d54-0f90-4324-a037-5811294f85e6","origin":"tls","data":{"error":{},"identifier":"paperless.zi0x.duckdns.org","issuers":["acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
Mar 03 13:18:37 revprox caddy[14584]: {"level":"error","ts":1709471917.856004,"logger":"tls.obtain","msg":"will retry","error":"[paperless.zi0x.duckdns.org] Obtain: [paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.222.19.97:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988335173) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":278.92926686,"max_duration":2592000}
Mar 03 13:20:37 revprox caddy[14584]: {"level":"info","ts":1709472037.8571117,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:20:37 revprox caddy[14584]: {"level":"debug","ts":1709472037.8576403,"logger":"events","msg":"event","name":"cert_obtaining","id":"9de1a4d8-4dac-4fc9-9637-a9d642175599","origin":"tls","data":{"identifier":"paperless.zi0x.duckdns.org"}}
Mar 03 13:20:38 revprox caddy[14584]: {"level":"info","ts":1709472038.518152,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:20:38 revprox caddy[14584]: {"level":"debug","ts":1709472038.7701044,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:21:03 revprox caddy[14584]: {"level":"debug","ts":1709472063.52169,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:21:05 revprox caddy[14584]: {"level":"error","ts":1709472065.5511808,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"paperless.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.223.106.16:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988382803) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Mar 03 13:21:05 revprox caddy[14584]: {"level":"debug","ts":1709472065.5513434,"logger":"events","msg":"event","name":"cert_failed","id":"1b58c019-4522-482e-802a-7af56f997ccf","origin":"tls","data":{"error":{},"identifier":"paperless.zi0x.duckdns.org","issuers":["acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
Mar 03 13:21:05 revprox caddy[14584]: {"level":"error","ts":1709472065.5514565,"logger":"tls.obtain","msg":"will retry","error":"[paperless.zi0x.duckdns.org] Obtain: [paperless.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.paperless.zi0x.duckdns.org\": dial tcp 15.223.106.16:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988382803) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":4,"retrying_in":300,"elapsed":426.624719207,"max_duration":2592000}
Mar 03 13:26:05 revprox caddy[14584]: {"level":"info","ts":1709472365.5786772,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:26:05 revprox caddy[14584]: {"level":"debug","ts":1709472365.57915,"logger":"events","msg":"event","name":"cert_obtaining","id":"ad113bce-6bcb-4d32-8f6e-58411c6ea288","origin":"tls","data":{"identifier":"paperless.zi0x.duckdns.org"}}
Mar 03 13:26:06 revprox caddy[14584]: {"level":"info","ts":1709472366.2400987,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:26:06 revprox caddy[14584]: {"level":"debug","ts":1709472366.4933376,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:26:11 revprox caddy[14584]: {"level":"debug","ts":1709472371.9537983,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:26:12 revprox caddy[14584]: {"level":"debug","ts":1709472372.1705382,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"paperless.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:26:13 revprox caddy[14584]: {"level":"info","ts":1709472373.3400533,"logger":"tls.issuance.acme.acme_client","msg":"authorization finalized","identifier":"paperless.zi0x.duckdns.org","authz_status":"valid"}
Mar 03 13:26:17 revprox caddy[14584]: {"level":"info","ts":1709472377.2417629,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:26:17 revprox caddy[14584]: {"level":"debug","ts":1709472377.242131,"logger":"events","msg":"event","name":"cert_obtained","id":"e62a818c-49e0-4171-a69d-2e6a37c5ae32","origin":"tls","data":{"certificate_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org/paperless.zi0x.duckdns.org.crt","identifier":"paperless.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","metadata_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org/paperless.zi0x.duckdns.org.json","private_key_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org/paperless.zi0x.duckdns.org.key","renewal":false,"storage_path":"certificates/acme-staging-v02.api.letsencrypt.org-directory/paperless.zi0x.duckdns.org"}}
Mar 03 13:26:17 revprox caddy[14584]: {"level":"info","ts":1709472377.2423437,"logger":"tls.obtain","msg":"releasing lock","identifier":"paperless.zi0x.duckdns.org"}
Mar 03 13:26:18 revprox caddy[14584]: {"level":"debug","ts":1709472378.3645875,"logger":"tls.cache","msg":"added certificate to cache","subjects":["paperless.zi0x.duckdns.org"],"expiration":1717244773,"managed":true,"issuer_key":"acme-staging-v02.api.letsencrypt.org-directory","hash":"3c92579dcc5e8916c090e4dad4868804e01659424d54051362164c05559bc65d","cache_size":3,"cache_capacity":10000}
Mar 03 13:26:18 revprox caddy[14584]: {"level":"debug","ts":1709472378.3650982,"logger":"events","msg":"event","name":"cached_managed_cert","id":"76cc9a54-8faf-4292-81b8-59e058db6dca","origin":"tls","data":{"sans":["paperless.zi0x.duckdns.org"]}}
What exactly is happening when Caddy is “waiting for solver”? I can see where it failed it waited for ~28s but in the last success it only waited ~5s.
It’s not always that amount of time though: this instance for nextcloud
took way longer to fail:
Mar 03 13:26:02 revprox caddy[14584]: {"level":"info","ts":1709472362.5413764,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"nextcloud.zi0x.duckdns.org"}
Mar 03 13:26:02 revprox caddy[14584]: {"level":"debug","ts":1709472362.5418456,"logger":"events","msg":"event","name":"cert_obtaining","id":"7bd5448b-59cf-4dc5-b13f-10818b3b402f","origin":"tls","data":{"identifier":"nextcloud.zi0x.duckdns.org"}}
Mar 03 13:26:03 revprox caddy[14584]: {"level":"info","ts":1709472363.2084017,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"nextcloud.zi0x.duckdns.org","challenge_type":"dns-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 03 13:26:03 revprox caddy[14584]: {"level":"debug","ts":1709472363.4638581,"logger":"tls.issuance.acme.acme_client","msg":"waiting for solver before continuing","identifier":"nextcloud.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:27:23 revprox caddy[14584]: {"level":"debug","ts":1709472443.603528,"logger":"tls.issuance.acme.acme_client","msg":"done waiting for solver","identifier":"nextcloud.zi0x.duckdns.org","challenge_type":"dns-01"}
Mar 03 13:27:24 revprox caddy[14584]: {"level":"error","ts":1709472444.0704942,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"nextcloud.zi0x.duckdns.org","issuer":"acme-staging-v02.api.letsencrypt.org-directory","error":"[nextcloud.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.nextcloud.zi0x.duckdns.org\": dial tcp 99.79.16.64:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988484613) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
Mar 03 13:27:24 revprox caddy[14584]: {"level":"debug","ts":1709472444.0705192,"logger":"events","msg":"event","name":"cert_failed","id":"318daa46-2865-44de-ae81-3aecc24f5c7b","origin":"tls","data":{"error":{},"identifier":"nextcloud.zi0x.duckdns.org","issuers":["acme-staging-v02.api.letsencrypt.org-directory"],"renewal":false}}
Mar 03 13:27:24 revprox caddy[14584]: {"level":"error","ts":1709472444.0705514,"logger":"tls.obtain","msg":"will retry","error":"[nextcloud.zi0x.duckdns.org] Obtain: [nextcloud.zi0x.duckdns.org] solving challenges: waiting for solver certmagic.solverWrapper to be ready: checking DNS propagation of \"_acme-challenge.nextcloud.zi0x.duckdns.org\": dial tcp 99.79.16.64:53: i/o timeout (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/138723713/14988484613) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":805.144746561,"max_duration":2592000}