Autohttps fails with Acme challenge

Help
1

1. The problem I’m having:

The same exact caddy config that has been working until recently stopped generating correct certificates. My certificates are now expired and I’m trying to fix that

2. Error messages and/or full log output:

Mar 06 16:02:55 jws-2 caddy[2320438]: caddy.HomeDir=/var/lib/caddy
Mar 06 16:02:55 jws-2 caddy[2320438]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Mar 06 16:02:55 jws-2 caddy[2320438]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Mar 06 16:02:55 jws-2 caddy[2320438]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Mar 06 16:02:55 jws-2 caddy[2320438]: caddy.Version=v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=
Mar 06 16:02:55 jws-2 caddy[2320438]: runtime.GOOS=linux
Mar 06 16:02:55 jws-2 caddy[2320438]: runtime.GOARCH=amd64
Mar 06 16:02:55 jws-2 caddy[2320438]: runtime.Compiler=gc
Mar 06 16:02:55 jws-2 caddy[2320438]: runtime.NumCPU=4
Mar 06 16:02:55 jws-2 caddy[2320438]: runtime.GOMAXPROCS=4
Mar 06 16:02:55 jws-2 caddy[2320438]: runtime.Version=go1.22.3
Mar 06 16:02:55 jws-2 caddy[2320438]: os.Getwd=/
Mar 06 16:02:55 jws-2 caddy[2320438]: LANG=en_US.UTF-8
Mar 06 16:02:55 jws-2 caddy[2320438]: LANGUAGE=en_US:
Mar 06 16:02:55 jws-2 caddy[2320438]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
Mar 06 16:02:55 jws-2 caddy[2320438]: NOTIFY_SOCKET=/run/systemd/notify
Mar 06 16:02:55 jws-2 caddy[2320438]: HOME=/var/lib/caddy
Mar 06 16:02:55 jws-2 caddy[2320438]: LOGNAME=caddy
Mar 06 16:02:55 jws-2 caddy[2320438]: USER=caddy
Mar 06 16:02:55 jws-2 caddy[2320438]: INVOCATION_ID=a95982fef59c4b22b8c0b59234a37821
Mar 06 16:02:55 jws-2 caddy[2320438]: JOURNAL_STREAM=9:96548210
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.555888,"msg":"using config from file","file":"/etc/caddy/Caddyfile"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5574236,"msg":"adapted config to JSON","adapter":"caddyfile"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"warn","ts":1741273375.5574415,"msg":"Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":12}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.558733,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.558928,"logger":"http.auto_https","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5589654,"logger":"http.auto_https","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5593889,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000440f00"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5597289,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5600598,"logger":"http.log","msg":"server running","name":"srv0","protocols":["h1","h2","h3"]}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5601277,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5601354,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["mia.jws.gmbh","rocket.jws.gmbh"]}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5612836,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Mar 06 16:02:55 jws-2 systemd[1]: Started Caddy.
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5623941,"logger":"tls.obtain","msg":"acquiring lock","identifier":"rocket.jws.gmbh"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5631394,"msg":"serving initial configuration"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5639021,"logger":"tls.obtain","msg":"lock acquired","identifier":"rocket.jws.gmbh"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5641725,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"rocket.jws.gmbh"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.564222,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/var/lib/caddy/.local/share/caddy","instance":"61da8da9-a356-4382-8918-a05642f6dae1","try_again":1741359775.5642192,"try_again_in":86399.99999951}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5643058,"logger":"tls","msg":"finished cleaning storage units"}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5651262,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["rocket.jws.gmbh"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5651681,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["rocket.jws.gmbh"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
Mar 06 16:02:55 jws-2 caddy[2320438]: {"level":"info","ts":1741273375.5651877,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1906443236","account_contact":[]}
Mar 06 16:02:56 jws-2 caddy[2320438]: {"level":"info","ts":1741273376.7431707,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Mar 06 16:03:07 jws-2 caddy[2320438]: {"level":"error","ts":1741273387.6156561,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]}}
Mar 06 16:03:07 jws-2 caddy[2320438]: {"level":"error","ts":1741273387.6157582,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/1906443236/360812448911","attempt":1,"max_attempts":3}
Mar 06 16:03:08 jws-2 caddy[2320438]: {"level":"error","ts":1741273388.7615025,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"rocket.jws.gmbh","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"rocket.jws.gmbh\" in the last 1h0m0s, retry after 2025-03-06 15:05:18 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account"}
Mar 06 16:03:08 jws-2 caddy[2320438]: {"level":"error","ts":1741273388.76199,"logger":"tls.obtain","msg":"will retry","error":"[rocket.jws.gmbh] Obtain: [rocket.jws.gmbh] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 429 urn:ietf:params:acme:error:rateLimited - too many failed authorizations (5) for \"rocket.jws.gmbh\" in the last 1h0m0s, retry after 2025-03-06 15:05:18 UTC: see https://letsencrypt.org/docs/rate-limits/#authorization-failures-per-hostname-per-account (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":13.197907799,"max_duration":2592000}
Mar 06 16:04:08 jws-2 caddy[2320438]: {"level":"info","ts":1741273448.7630234,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"rocket.jws.gmbh"}
Mar 06 16:04:08 jws-2 caddy[2320438]: {"level":"info","ts":1741273448.764271,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/168163553","account_contact":[]}
Mar 06 16:04:10 jws-2 caddy[2320438]: {"level":"info","ts":1741273450.017542,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:04:31 jws-2 caddy[2320438]: {"level":"error","ts":1741273471.6512516,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/lyvU-DDxRx-wamkFtYG9jLUS1lVRueGPoUWv3eNfhcg: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
Mar 06 16:04:31 jws-2 caddy[2320438]: {"level":"error","ts":1741273471.6513526,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/lyvU-DDxRx-wamkFtYG9jLUS1lVRueGPoUWv3eNfhcg: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068640394","attempt":1,"max_attempts":3}
Mar 06 16:04:32 jws-2 caddy[2320438]: {"level":"info","ts":1741273472.9914489,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:04:44 jws-2 caddy[2320438]: {"level":"error","ts":1741273484.0840743,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]}}
Mar 06 16:04:44 jws-2 caddy[2320438]: {"level":"error","ts":1741273484.0841553,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068648334","attempt":2,"max_attempts":3}
Mar 06 16:04:44 jws-2 caddy[2320438]: {"level":"error","ts":1741273484.084209,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"rocket.jws.gmbh","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error"}
Mar 06 16:04:44 jws-2 caddy[2320438]: {"level":"error","ts":1741273484.0842795,"logger":"tls.obtain","msg":"will retry","error":"[rocket.jws.gmbh] Obtain: [rocket.jws.gmbh] solving challenge: rocket.jws.gmbh: [rocket.jws.gmbh] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":108.520198063,"max_duration":2592000}
Mar 06 16:06:44 jws-2 caddy[2320438]: {"level":"info","ts":1741273604.0889757,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"rocket.jws.gmbh"}
Mar 06 16:06:44 jws-2 caddy[2320438]: {"level":"info","ts":1741273604.0938933,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/168163553","account_contact":[]}
Mar 06 16:06:44 jws-2 caddy[2320438]: {"level":"info","ts":1741273604.577091,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:07:06 jws-2 caddy[2320438]: {"level":"error","ts":1741273626.0076258,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/vPRA5uZgTqp1-3zX0SzkUqDmbHK50DYee3xNYKlAstY: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
Mar 06 16:07:06 jws-2 caddy[2320438]: {"level":"error","ts":1741273626.0077863,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/vPRA5uZgTqp1-3zX0SzkUqDmbHK50DYee3xNYKlAstY: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068691934","attempt":1,"max_attempts":3}
Mar 06 16:07:07 jws-2 caddy[2320438]: {"level":"info","ts":1741273627.3472955,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:07:18 jws-2 caddy[2320438]: {"level":"error","ts":1741273638.574617,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]}}
Mar 06 16:07:18 jws-2 caddy[2320438]: {"level":"error","ts":1741273638.5747054,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068698714","attempt":2,"max_attempts":3}
Mar 06 16:07:18 jws-2 caddy[2320438]: {"level":"error","ts":1741273638.5747848,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"rocket.jws.gmbh","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error"}
Mar 06 16:07:18 jws-2 caddy[2320438]: {"level":"error","ts":1741273638.5749168,"logger":"tls.obtain","msg":"will retry","error":"[rocket.jws.gmbh] Obtain: [rocket.jws.gmbh] solving challenge: rocket.jws.gmbh: [rocket.jws.gmbh] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":3,"retrying_in":120,"elapsed":263.010829018,"max_duration":2592000}
Mar 06 16:09:18 jws-2 caddy[2320438]: {"level":"info","ts":1741273758.5776286,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"rocket.jws.gmbh"}
Mar 06 16:09:18 jws-2 caddy[2320438]: {"level":"info","ts":1741273758.5790088,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/168163553","account_contact":[]}
Mar 06 16:09:19 jws-2 caddy[2320438]: {"level":"info","ts":1741273759.0674791,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:09:40 jws-2 caddy[2320438]: {"level":"error","ts":1741273780.3680968,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/ZIBINyiAsBWZGUySMAmpVxZ326-HZOCg5KPSPHVOQk0: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
Mar 06 16:09:40 jws-2 caddy[2320438]: {"level":"error","ts":1741273780.3682444,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/ZIBINyiAsBWZGUySMAmpVxZ326-HZOCg5KPSPHVOQk0: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068739574","attempt":1,"max_attempts":3}
Mar 06 16:09:41 jws-2 caddy[2320438]: {"level":"info","ts":1741273781.704288,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:09:52 jws-2 caddy[2320438]: {"level":"error","ts":1741273792.8255138,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]}}
Mar 06 16:09:52 jws-2 caddy[2320438]: {"level":"error","ts":1741273792.8256025,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068745974","attempt":2,"max_attempts":3}
Mar 06 16:09:52 jws-2 caddy[2320438]: {"level":"error","ts":1741273792.825659,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"rocket.jws.gmbh","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error"}
Mar 06 16:09:52 jws-2 caddy[2320438]: {"level":"error","ts":1741273792.825766,"logger":"tls.obtain","msg":"will retry","error":"[rocket.jws.gmbh] Obtain: [rocket.jws.gmbh] solving challenge: rocket.jws.gmbh: [rocket.jws.gmbh] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":4,"retrying_in":300,"elapsed":417.261684559,"max_duration":2592000}
Mar 06 16:14:52 jws-2 caddy[2320438]: {"level":"info","ts":1741274092.8274183,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"rocket.jws.gmbh"}
Mar 06 16:14:52 jws-2 caddy[2320438]: {"level":"info","ts":1741274092.8314815,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/168163553","account_contact":[]}
Mar 06 16:14:53 jws-2 caddy[2320438]: {"level":"info","ts":1741274093.8975346,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:15:15 jws-2 caddy[2320438]: {"level":"error","ts":1741274115.2491705,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/2_B2Og0DJzO8BhZwd5g0i6GnnTfU4Cv36yHv3d9rusU: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
Mar 06 16:15:15 jws-2 caddy[2320438]: {"level":"error","ts":1741274115.2493062,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/2_B2Og0DJzO8BhZwd5g0i6GnnTfU4Cv36yHv3d9rusU: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068841724","attempt":1,"max_attempts":3}
Mar 06 16:15:16 jws-2 caddy[2320438]: {"level":"info","ts":1741274116.578665,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:15:27 jws-2 caddy[2320438]: {"level":"error","ts":1741274127.5386288,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]}}
Mar 06 16:15:27 jws-2 caddy[2320438]: {"level":"error","ts":1741274127.5387325,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23068849454","attempt":2,"max_attempts":3}
Mar 06 16:15:27 jws-2 caddy[2320438]: {"level":"error","ts":1741274127.5388646,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"rocket.jws.gmbh","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error"}
Mar 06 16:15:27 jws-2 caddy[2320438]: {"level":"error","ts":1741274127.5389743,"logger":"tls.obtain","msg":"will retry","error":"[rocket.jws.gmbh] Obtain: [rocket.jws.gmbh] solving challenge: rocket.jws.gmbh: [rocket.jws.gmbh] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":5,"retrying_in":600,"elapsed":751.974887181,"max_duration":2592000}
Mar 06 16:25:27 jws-2 caddy[2320438]: {"level":"info","ts":1741274727.5402913,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"rocket.jws.gmbh"}
Mar 06 16:25:27 jws-2 caddy[2320438]: {"level":"info","ts":1741274727.5435512,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/168163553","account_contact":[]}
Mar 06 16:25:28 jws-2 caddy[2320438]: {"level":"info","ts":1741274728.6399972,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:25:49 jws-2 caddy[2320438]: {"level":"error","ts":1741274749.8388948,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/7ZyNcNVSetYtDVlfR6i8T6mhyGdmRjGqsjv6PtGR4sM: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
Mar 06 16:25:49 jws-2 caddy[2320438]: {"level":"error","ts":1741274749.8390043,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/7ZyNcNVSetYtDVlfR6i8T6mhyGdmRjGqsjv6PtGR4sM: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23069040274","attempt":1,"max_attempts":3}
Mar 06 16:25:51 jws-2 caddy[2320438]: {"level":"info","ts":1741274751.1827536,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:26:02 jws-2 caddy[2320438]: {"level":"error","ts":1741274762.2674158,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]}}
Mar 06 16:26:02 jws-2 caddy[2320438]: {"level":"error","ts":1741274762.267809,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23069046514","attempt":2,"max_attempts":3}
Mar 06 16:26:02 jws-2 caddy[2320438]: {"level":"error","ts":1741274762.2679796,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"rocket.jws.gmbh","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error"}
Mar 06 16:26:02 jws-2 caddy[2320438]: {"level":"error","ts":1741274762.2681298,"logger":"tls.obtain","msg":"will retry","error":"[rocket.jws.gmbh] Obtain: [rocket.jws.gmbh] solving challenge: rocket.jws.gmbh: [rocket.jws.gmbh] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":6,"retrying_in":600,"elapsed":1386.704029861,"max_duration":2592000}
Mar 06 16:36:02 jws-2 caddy[2320438]: {"level":"info","ts":1741275362.2721272,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"rocket.jws.gmbh"}
Mar 06 16:36:02 jws-2 caddy[2320438]: {"level":"info","ts":1741275362.2802463,"logger":"http","msg":"using ACME account","account_id":"https://acme-staging-v02.api.letsencrypt.org/acme/acct/168163553","account_contact":[]}
Mar 06 16:36:03 jws-2 caddy[2320438]: {"level":"info","ts":1741275363.4058702,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:36:24 jws-2 caddy[2320438]: {"level":"error","ts":1741275384.682444,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/xeGOs9drdeNMZkt5BTTdWaegBp_unS6xkS8c9OLnuYM: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]}}
Mar 06 16:36:24 jws-2 caddy[2320438]: {"level":"error","ts":1741275384.6825566,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:connection","title":"","detail":"156.67.25.163: Fetching https://rocket.jws.gmbh/.well-known/acme-challenge/xeGOs9drdeNMZkt5BTTdWaegBp_unS6xkS8c9OLnuYM: Timeout during connect (likely firewall problem)","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23069248384","attempt":1,"max_attempts":3}
Mar 06 16:36:26 jws-2 caddy[2320438]: {"level":"info","ts":1741275386.022283,"logger":"http.acme_client","msg":"trying to solve challenge","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Mar 06 16:36:37 jws-2 caddy[2320438]: {"level":"error","ts":1741275397.143282,"logger":"http.acme_client","msg":"challenge failed","identifier":"rocket.jws.gmbh","challenge_type":"tls-alpn-01","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]}}
Mar 06 16:36:37 jws-2 caddy[2320438]: {"level":"error","ts":1741275397.143536,"logger":"http.acme_client","msg":"validating authorization","identifier":"rocket.jws.gmbh","problem":{"type":"urn:ietf:params:acme:error:tls","title":"","detail":"156.67.25.163: remote error: tls: internal error","instance":"","subproblems":[]},"order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/168163553/23069255334","attempt":2,"max_attempts":3}
Mar 06 16:36:37 jws-2 caddy[2320438]: {"level":"error","ts":1741275397.1436605,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"rocket.jws.gmbh","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error"}
Mar 06 16:36:37 jws-2 caddy[2320438]: {"level":"error","ts":1741275397.143778,"logger":"tls.obtain","msg":"will retry","error":"[rocket.jws.gmbh] Obtain: [rocket.jws.gmbh] solving challenge: rocket.jws.gmbh: [rocket.jws.gmbh] authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - 156.67.25.163: remote error: tls: internal error (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":7,"retrying_in":600,"elapsed":2021.57967253,"max_duration":2592000}

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

running natively with systemd
linux ubuntu VPS

b. Command:

restart caddy

sudo systemctl restart caddy

reload config (in correct dir)

sudo caddy reload

c. Service/unit/compose file:

PASTE OVER THIS, BETWEEN THE ``` LINES.
Please use the preview pane to ensure it looks nice.

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

rocket.jws.gmbh {
	reverse_proxy localhost:8267
}


mia.jws.gmbh {
	reverse_proxy localhost:8008
}

5. Links to relevant resources:

2

Inside your error, the following URL is mentioned:

trying to reach this URL shows a connection time out, just like the error report.

Note that letsencrypt is not compatible with region blocking by IP in the firewall, it can select any datacenter at random and connect to your server

Note that rocket.jws.gmbh resolves to 2a02:c206:3014:8467::1 and 156.67.25.163. Lets encrypt tries them in order, IPv6 first, then IPv4 if IPv6 is unreachable. There is a maximum total timeout.

In your case, you have an issue at 2a02:c206:3014:8467::1, which just times out every connection

In a command line on a dual stack host, test it with:

$ curl -vsS --insecure -m 10 --happy-eyeballs-timeout-ms 10000 \
https://rocket.jws.gmbh/.well-known/acme-challenge/7ZyNcNVSetYtDVlfR6i8T6mhyGdmRjGqsjv6PtGR4sM > /dev/null

(Note that Letsencrypt only does 1 fallback to IPv4 if IPv6 times out with the HTTP challenge, this is wasted for the HTTP to HTTPS redirect, see IPv6 Support - Let's Encrypt)

2 Likes