Certbot issues: How can I identify them?

Louen · August 4, 2024, 10:17pm

I’m having issues with certbot, but I cannot clearly see what’s going on

I don’t know if there is an issue with the domain, with the DNS propagation when I change DNS, if there is a rate limitation, etc…

I’m running Caddy over docker

For example, for one domain this is all I get in docker logs

{"level":"info","ts":1722806984.2762606,"logger":"tls.obtain","msg":"releasing lock","identifier":"example.com"}
{"level":"error","ts":1722806984.2774055,"logger":"tls","msg":"job failed","error":"example.com: obtaining certificate: [example.com] Obtain: [example.com] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)"}

From there, I cannot really understand what’s going on.

Is there a way to get the exact information?

Bruce5051 · August 4, 2024, 10:45pm

Hi @Louen,

Cetbot information can be found here Welcome to the Certbot documentation! — Certbot 2.12.0.dev0 documentation

Also retry the Cerbot command you use with adding the -vvv option that gives lots of verbose information.

And make sure you are running Certbot with sufficient privileges (i.e. usually sudo certbot . . .

Louen · August 5, 2024, 9:08am

I’m not using any certbot command, everything is being handled automatically (and for not, not sure how exactly)

Mohammed90 · August 5, 2024, 9:38am

Caddy doesn’t use certbot. Certbot is a separate software that isn’t affiliated nor used by Caddy. We have our own ACME client implementation. Please fill out the full template so we can help you. There are many reasons why that message might be in the logs,

Louen · August 5, 2024, 11:04am

I have filed a bug because it seems that it is

github.com/caddyserver/caddy

Weird broken loop with 308 Permanent Redirect on php_fastcgi

opened 10:53AM - 05 Aug 24 UTC

SirLouen

I've noted that `php_fastcgi` directive has a default behaviour that can complet…ely break the Let's Encrypt retrieval process I have this test domain: fedora.mcamargo.es Configured with this Caddyfile ``` fedora.mcamargo.es { root * /usr/share/caddy/fedora file_server php_fastcgi unix//run/php/php-fpm.sock { root /var/www/html } } ``` I'm using a dockerized Caddy (caddy:latest 2.8.4) with an another dockerized PHP-FPM server At some point I deleted the volume where certs were stored. As we can see here this is the response: ``` curl http://fedora.mcamargo.es -v * Host fedora.mcamargo.es:80 was resolved. * IPv6: (none) * IPv4: 135.181.80.106 * Trying 135.181.80.106:80... * Connected to fedora.mcamargo.es (135.181.80.106) port 80 > GET / HTTP/1.1 > Host: fedora.mcamargo.es > User-Agent: curl/8.7.1 > Accept: */* > * Request completely sent off < HTTP/1.1 308 Permanent Redirect < Connection: close < Location: https://fedora.mcamargo.es/ < Server: Caddy < Date: Mon, 05 Aug 2024 10:47:13 GMT < Content-Length: 0 < * Closing connection ``` There is a permanent 308 redirect that inhibits the recreation of the new certificate, just as we can see in the letsdebug error: https://letsdebug.net/fedora.mcamargo.es/2155062 I'm going to research how I can stop that 308 redirect to HTTPS to re-issue the certificates, but the system is broken as-is and I have not idea for now how to sort this out. I'm not sure if this issue is related somehow: https://github.com/caddyserver/caddy/issues/5660 as I cannot relate to the content of it, but maybe there is some relationship If I change the hostname in the Caddyfile to ``` http://fedora.mcamargo.es { root * /usr/share/caddy/fedora file_server php_fastcgi unix//run/php/php-fpm.sock { root /var/www/html } } ``` As we can see here, it's fully compliant again. https://letsdebug.net/fedora.mcamargo.es/2155077 And the site works well (without HTTPS) That pesky 308 is driving me nuts :)

These are the full logs

l":"info","ts":1722856744.7359416,"logger":"tls.obtain","msg":"lock acquired","identifier":"fedora.mcamargo.es"}
{"level":"info","ts":1722856744.7362647,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"fedora.mcamargo.es"}
{"level":"info","ts":1722856744.7378972,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1873678947","account_contact":[]}
{"level":"info","ts":1722856744.7379506,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["fedora.mcamargo.es"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"info","ts":1722856744.7380552,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["fedora.mcamargo.es"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
{"level":"error","ts":1722856745.4084947,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"fedora.mcamargo.es","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"error","ts":1722856745.4086013,"logger":"tls.obtain","msg":"will retry","error":"[fedora.mcamargo.es] Obtain: [fedora.mcamargo.es] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.672511664,"max_duration":2592000}

Something is happening, and I cannot identify it

If I set in the config

http://fedora.mcamargo.es, https://fedora.mcamargo.es {
...
}

http://fedora.mcamargo.es works well (without certificate)

But if I set:

{
        auto_https disable_redirects
}

fedora.mcamargo.es {
...
}

http://fedora.mcamargo.es is unreachable (I’m assuming that no protocol in Caddy means https by default, so nothing is bind to http)

Mohammed90 · August 5, 2024, 12:21pm

Why are you using disable_redirects? If you’re not using any other global option, use the following format:

fedora.mcamargo.es {
...
}

That’s all that’s needed. It works for everyone else.

Louen · August 5, 2024, 12:36pm

I’m not using that specifically

I’ve tried like 10 things. This is just one among all those 10 things just to see that it doesn’t work either.

Obviously I started with, that was the first thing I tried…

fedora.mcamargo.es {
...
}

At this point, nothing works, I’m trying to see if I can get a little bit of more info in the logs or something because I’m 100% clueless of what’s going on…

I’ve added

{
  debug
}

To add more verbosity

caddy  | {"level":"info","ts":1722861074.6596396,"msg":"serving initial configuration"}
caddy  | {"level":"info","ts":1722861074.6600025,"logger":"tls.obtain","msg":"acquiring lock","identifier":"fedora.mcamargo.es"}
caddy  | {"level":"info","ts":1722861074.6817706,"logger":"tls.obtain","msg":"lock acquired","identifier":"fedora.mcamargo.es"}
caddy  | {"level":"info","ts":1722861074.682164,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"fedora.mcamargo.es"}
caddy  | {"level":"debug","ts":1722861074.6823566,"logger":"events","msg":"event","name":"cert_obtaining","id":"9e9cf94d-9f26-477d-a4cb-c5279811dc1b","origin":"tls","data":{"identifier":"fedora.mcamargo.es"}}
caddy  | {"level":"debug","ts":1722861074.682894,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"acme-v02.api.letsencrypt.org-directory"}
caddy  | {"level":"info","ts":1722861074.6829708,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data/caddy","instance":"7bec2d31-f920-4348-b44a-fd5d01d128eb","try_again":1722947474.6829655,"try_again_in":86399.999999185}
caddy  | {"level":"info","ts":1722861074.6831014,"logger":"tls","msg":"finished cleaning storage units"}
caddy  | {"level":"info","ts":1722861074.683595,"logger":"http","msg":"waiting on internal rate limiter","identifiers":["fedora.mcamargo.es"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy  | {"level":"info","ts":1722861074.683631,"logger":"http","msg":"done waiting on internal rate limiter","identifiers":["fedora.mcamargo.es"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy  | {"level":"info","ts":1722861074.68369,"logger":"http","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1873678947","account_contact":[]}
caddy  | {"level":"debug","ts":1722861075.1663668,"logger":"http.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["746"],"Content-Type":["application/json"],"Date":["Mon, 05 Aug 2024 12:31:15 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddy  | {"level":"debug","ts":1722861075.1668687,"logger":"http.acme_client","msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1873678947","identifiers":["fedora.mcamargo.es"]}
caddy  | {"level":"debug","ts":1722861075.326039,"logger":"http.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 05 Aug 2024 12:31:15 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["QYxeyab8DjxRsHH1sLA6jFmfoLE8S08iXhQDHfAjESgEfpn3OyE"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
caddy  | {"level":"debug","ts":1722861075.4871967,"logger":"http.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1873678947"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["107"],"Content-Type":["application/problem+json"],"Date":["Mon, 05 Aug 2024 12:31:15 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["VbHcGKwn7lYkfnUCTLwWWACiemiMrVlu3OWCc8QcEvD_1iiDdlY"],"Server":["nginx"]},"status_code":400}
caddy  | {"level":"error","ts":1722861075.4873748,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"fedora.mcamargo.es","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
caddy  | {"level":"debug","ts":1722861075.4875069,"logger":"events","msg":"event","name":"cert_failed","id":"0162d9bf-fed4-41e7-aa54-e9250562d8e7","origin":"tls","data":{"error":{},"identifier":"fedora.mcamargo.es","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
caddy  | {"level":"error","ts":1722861075.4875762,"logger":"tls.obtain","msg":"will retry","error":"[fedora.mcamargo.es] Obtain: [fedora.mcamargo.es] creating new order: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-order: HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.805740177,"max_duration":2592000}

But still I cannot see anything relevant in the logs.

Bruce5051 · August 5, 2024, 4:29pm

Yet this Post’s Title is “Certbot issues: How can I identify them?”

Louen · August 5, 2024, 5:00pm

Yes, I’ve been able to “solve” the issue, but I don’t really know whats happening
+
I have not solved it thanks to the logs

Which leaves me a little worried.

system · September 4, 2024, 5:00pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.