ACME Challenge Failing - Caddy not handling http request?

1. Caddy version (caddy version):

2.4.6

2. How I run Caddy:

a. System environment:

Docker

b. Command:

caddy run --config /dockerapp/caddy/Caddyfile

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

{
  storage redis {
    host {$REDIS_HOST}
  }
  on_demand_tls {
    ask https://www.my-domain.org/custom-domain-check
    interval 2m
    burst 5
  }
  debug
}

(SecurityHeaders) {
  header_up X-Real-IP {remote_host}
  header_up X-Forwarded-Proto {scheme}
}


my-domain.org, *.my-domain.org {

    @notStatic {
        not file
    }
    reverse_proxy @notStatic web:3000
    request_body {
      max_size 100MB
    }
    log {
        output stdout
    }
    tls me@my-domain.org {
        dns route53
    }
}


:443, :80 {
    @notStatic {
        not file
    }
    reverse_proxy @notStatic web:3000
    request_body {
      max_size 100MB
    }
    tls {
        on_demand
    }
}

3. The problem I’m having:

I’m using Caddy on instances behind a load balancer, to generate on demand tls for a website we host on a client’s custom domain, my-client.org. I tried to write my Caddyfile so that on demand certs are stored in redis, but it’s unclear to me if that’s working since it’s repeatedly “asking” caddy to generate on demand tls for my-client.org.

Visiting my-client.org in the browser, everything appears to work fine - Caddy issues a valid certificate and the page loads over https. But when I look at the Caddy logs, it’s full of errors about reaching rate limits, could not get certificate from issuer, etc (log below). Furthermore, in my web app’s production logs, I see tons of 404 errors for GET requests to https://my-client.org/.well-known/acme-challenge/****

So https://my-client.org appears to be working fine, but the Caddy logs and acme-challenge 404s are telling me that something must be misconfigured. Does anyone have any insights into what’s going wrong and how to fix it?

4. Error messages and/or full log output:


caddy_1    | {"level":"info","ts":1641181016.9673479,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["my-client.org"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
 
caddy_1    | {"level":"error","ts":1641181017.2001925,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"my-client.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/"}
 
caddy_1    | {"level":"warn","ts":1641181017.2008576,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
 
caddy_1    | {"level":"info","ts":1641181017.86142,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"Wu_IeQsno34jbOSPxDPWwg"}
 
caddy_1    | {"level":"info","ts":1641181027.7515316,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["my-client.org"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
 
caddy_1    | {"level":"info","ts":1641181027.7515597,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["my-client.org"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
 
03:37:12.238
caddy_1    | {"level":"info","ts":1641181032.2379267,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"my-client.org","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
 
caddy_1    | {"level":"info","ts":1641181090.0311844,"logger":"tls.on_demand","msg":"obtaining new certificate","server_name":"my-client.org"}

caddy_1    | {"level":"info","ts":1641181090.0329618,"logger":"tls.obtain","msg":"acquiring lock","identifier":"my-client.org"}
 
caddy_1    | {"level":"warn","ts":1641181194.9624903,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/authz/OUSc4KaAYokoSmloWA9keA","error":"performing request: Post \"https://acme.zerossl.com/v2/DV90/authz/OUSc4KaAYokoSmloWA9keA\": context deadline exceeded"}
 
caddy_1    | {"level":"error","ts":1641181194.9625244,"logger":"tls.issuance.acme.acme_client","msg":"deactivating authorization","identifier":"my-client.org","authz":"https://acme.zerossl.com/v2/DV90/authz/OUSc4KaAYokoSmloWA9keA","error":"attempt 1: https://acme.zerossl.com/v2/DV90/authz/OUSc4KaAYokoSmloWA9keA: context deadline exceeded"}
 
caddy_1    | {"level":"error","ts":1641181194.9625866,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"my-client.org","issuer":"acme.zerossl.com-v2-DV90","error":"[my-client.org] solving challenges: [my-client.org] context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/CZXYnA-1DDPTqGkw3i4ZnQ) (ca=https://acme.zerossl.com/v2/DV90)"}
 
caddy_1    | {"level":"error","ts":1641181194.9625986,"logger":"tls.obtain","msg":"will retry","error":"[my-client.org] Obtain: [my-client.org] solving challenges: [my-client.org] context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/CZXYnA-1DDPTqGkw3i4ZnQ) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":177.99758203,"max_duration":2592000}
 

5. What I already tried:

Reading through documentation and threads in the Caddy forum.

6. Links to relevant resources:

This means that Let’s Encrypt started rate limiting you.

I’m thinking your storage isn’t working properly. You build Caddy with the GitHub - gamalan/caddy-tlsredis: Redis Storage using for Caddy TLS Data plugin, right?

What does your Dockerfile look like? How about your docker-compose.yml or Docker commands you use to run Caddy?

What’s the value you have for {$REDIS_HOST}? If it has a port number in it, you probably want to use the address option instead of host (see the README for the redis plugin).

You didn’t completely fill out the help topic template so there’s still some question marks here.

2 Likes

Yes, I’m building Caddy with gamalan/caddy-tlsredis. To try and narrow down what the issue is, I hard-coded my redis address into the Caddyfile instead of using {$REDIS_HOST}, and I think that’s helped illuminate the issue a bit more. I can post my docker-compose.yml file if that’d be helpful, but I think the issue is elsewhere after looking into it some more.

I think the issue may have to do with the fact that my app redirects all requests to naked domain my-client.org to www.my-client.org. I’m doing this because I need www.my-client.org to be the canonical URL for the page, not the naked domain.

Looking at the Caddy logs with debug enabled, I see the following successful tls cache lookup for www.my-client.org:

caddy_1    | {"level":"debug","ts":1641240982.5736578,"logger":"tls.handshake","msg":"choosing certificate","identifier":"www.my-client.org","num_choices":1}
 
caddy_1    | {"level":"debug","ts":1641240982.5736866,"logger":"tls.handshake","msg":"default certificate selection results","identifier":"www.my-client.org","subjects":["www.my-client.org"],"managed":true,"issuer_key":"acme-v02.api.letsencrypt.org-directory","hash":"93baf7aca08d35f755a33dcc5e67e944be7ffd9080df32048d00e039f5527aor3"}

caddy_1    | {"level":"debug","ts":1641240982.5736947,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["www.my-client.org"],"managed":true,"expiration":1648352583,"hash":"93baf7aca08d35f755a33dcc5e67e944be7ffd9080df32048d00e039f5527aor3"}

But it seems unable to cache the certificate for the naked domain my-client.org. I should specify that the ask endpoint is returning 200 status codes for both my-client.org and www.my-client.org. Here are the Caddy logs I’m seeing for naked domain requests:

caddy_1    | {"level":"debug","ts":1641241483.4836013,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"my-client.org"}
 
caddy_1    | {"level":"debug","ts":1641241483.4836266,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.org"}
 
caddy_1    | {"level":"debug","ts":1641241483.4836302,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}

caddy_1    | {"level":"debug","ts":1641241499.5190558,"logger":"http.stdlib","msg":"http: TLS handshake error from 91.199.212.132:39006: timed out waiting to obtain certificate for my-client.org"}

caddy_1    | {"level":"error","ts":1641241734.093426,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"my-client.org","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many failed authorizations recently: see https://letsencrypt.org/docs/rate-limits/"}

caddy_1    | {"level":"debug","ts":1641241734.09344,"logger":"tls.obtain","msg":"trying issuer 2/2","issuer":"acme.zerossl.com-v2-DV90"}
 
caddy_1    | {"level":"warn","ts":1641241734.0942879,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
 
caddy_1    | {"level":"info","ts":1641241734.688482,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"*****-rBg"}

caddy_1    | {"level":"info","ts":1641241735.5952907,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["my-client.org"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
 
caddy_1    | {"level":"info","ts":1641241735.5953114,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["my-client.org"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}

I’m still seeing 404 errors in my web app for http requests to http://my-client.org/.well-known/acme-challenge/rbM0HKBAQNRTT****. And after digging through these, I can confirm that these 404s are all originating from http requests to the naked domain my-client.org (note: my web app automatically redirects these to www.my-client.org).

I understand the problem better now, but am still unsure about how to proceed. Shouldn’t Caddy be able to verify and cache the tls for the naked domain as well? Might I have something misconfigured somewhere?

If you have direct access to your Redis instance, try and see if you can find a certificate for that domain in the storage.

Caddy may have never actually issued one according to these logs.

It seems like you’re still rate limited by Let’s Encrypt, probably due to trying too many times with non-functional storage maybe.

And the logs end before the attempt to issue with ZeroSSL completes.

The 404s you see in your app are because Caddy received a challenge request from one of the issuers, without knowing the answer for that challenge, i.e. the challenge is no longer in storage. So it falls through and passes it through to the request handlers for your :80 site you have configured in case you might have some ACME client upstream (you don’t).

I don’t feel like I have a full picture of what’s going on – one reason being that the domain names are being omitted so I can’t do some DNS queries to ensure that’s correct as well (but I don’t see any evidence that it might be from what you said tbh).

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.