Caddy SSL renewal failure

1. The problem I’m having:

I run several subdomains through reverse proxy and Caddy. At some point my SSL certificates stopped being renewed and expired. Reading through the logs, it seems like it’s failing to renew through ZeroSSL. Any help decoding the error messages is greatly appreciated!

p.s. when using reverse_proxy, is it necessary to include http://?

2. Error messages and/or full log output:

I’ve pulled error messages for just one of the sub-domains (I think I got it all) since the logs are the same for the various subdomains and too large to copy & paste into the topic. The complete logs are here.

{"level":"info","ts":1695153826.9741719,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["sonarr.greghao.com","radarr.greghao.com","trans.greghao.com","tautulli.greghao.com","jackett.greghao.com","www.greghao.com","sabnzbd.greghao.com","adguard.greghao.com","overseerr.greghao.com"]}
{"level":"warn","ts":1695153827.053664,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [radarr.greghao.com]: parsing OCSP response: ocsp: error from server: unauthorized","identifiers":["radarr.greghao.com"]}
{"level":"info","ts":1695153827.0541282,"logger":"tls.renew","msg":"acquiring lock","identifier":"radarr.greghao.com"}
{"level":"info","ts":1695153827.057894,"logger":"tls.renew","msg":"lock acquired","identifier":"radarr.greghao.com"}
{"level":"info","ts":1695153827.0585961,"logger":"tls.renew","msg":"renewing certificate","identifier":"radarr.greghao.com","remaining":-99803.058595}
{"level":"info","ts":1695153827.6519349,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"radarr.greghao.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
{"level":"error","ts":1695153858.954658,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"radarr.greghao.com","challenge_type":"http-01","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2606:4700:3036::ac43:d0c9: Invalid response from http://radarr.greghao.com/.well-known/acme-challenge/LlLSXxgUWCqyeGTgw1XRz1rdYYg3KboPI0-ZtkTkScU: 522","instance":"","subproblems":[]}}
{"level":"error","ts":1695153858.9547222,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"radarr.greghao.com","problem":{"type":"urn:ietf:params:acme:error:unauthorized","title":"","detail":"2606:4700:3036::ac43:d0c9: Invalid response from http://radarr.greghao.com/.well-known/acme-challenge/LlLSXxgUWCqyeGTgw1XRz1rdYYg3KboPI0-ZtkTkScU: 522","instance":"","subproblems":[]},"order":"https://acme-v02.api.letsencrypt.org/acme/order/974460856/209505349976","attempt":1,"max_attempts":3}

3. Caddy version:

V 2.7.4

4. How I installed and ran Caddy:

brew install caddy

a. System environment:

MacOS 13.5.2

b. Command:

brew services start caddy

d. My complete Caddy config:

{
        email greg@twilightstar.net
}
(basic-auth) {
        basicauth / {
                1234 abcd
        }
}

sabnzbd.greghao.com {
        reverse_proxy http://127.0.0.1:8080
}

trans.greghao.com {
        reverse_proxy http://127.0.0.1:9091
}

radarr.greghao.com {
        import basic-auth
        reverse_proxy http://127.0.0.1:9092
}

sonarr.greghao.com {
        import basic-auth
        reverse_proxy http://127.0.0.1:9093
}

tautulli.greghao.com {
        reverse_proxy http://127.0.0.1:9094
}

www.greghao.com {
        import basic-auth
        reverse_proxy http://127.0.0.1:9095
}

jackett.greghao.com {
        import basic-auth
        reverse_proxy http://127.0.0.1:9096
}

overseerr.greghao.com {
        reverse_proxy http://127.0.0.1:9097
}

adguard.greghao.com {
        reverse_proxy http://192.168.10.52
}

5. Links to relevant resources:

Did you turn on Cloudflare proxying within the past ~3 months?

If Cloudflare is proxying, then HTTP requests don’t reach Caddy, so the ACME HTTP challenge fails.

You’ll either need to turn off Cloudflare’s proxy (orange cloud → grey cloud) or turn off HTTP->HTTPS redirects in Cloudflare (I don’t use it myself so I’m not sure exactly where that option is, but it should be in Rules or something like that, maybe called “Always Use HTTPS”).

So, I do have Cloudflare proxy enabled (although as far as I recall it’s always been enabled). But while digging around to turning off HTTP->HTTPS redirects, I noticed that Cloudflare has already created a wildcard SSL cert via Let’s Encrypt. Which leads me to wonder, is it possible to run Caddy without having it trying to pull a cert since I can get it through Cloudflare?

And for record keeping, the redirect that @francislavoie is referring to is located at:
Cloudflare Dashboard → SSL/TLS → Edge Certificate under “Always Use HTTPS”.

Update: I have turned off the “Always Use HTTPS” redirect but it looks like Caddy is still failing. Do I need to also modify this from “Full (strict)” to “Off”? New logs are here

Looks like requests on port 80 are timing out:

$ curl -v http://overseerr.greghao.com/                                                                           
*   Trying 172.64.80.1:80...
* Connected to overseerr.greghao.com (172.64.80.1) port 80 (#0)
> GET / HTTP/1.1
> Host: overseerr.greghao.com
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 522 
< Date: Tue, 19 Sep 2023 23:45:25 GMT
< Content-Length: 0
< Connection: keep-alive
< Cache-Control: no-store, no-cache
< CF-Cache-Status: DYNAMIC
< Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=oWuWWWjdvMx3LrtOjj835X8NRTPQW1tUkIDnrtPnIm0EYJrqLnFVCjGrNugB0BsXPa0JS3r1jQTji8Kd7m%2Fn%2FLBADU01npfa1nazColNc1avAXmVyDAejI2IQvyYeOubgmSjq1g2%2FhE%3D"}],"group":"cf-nel","max_age":604800}
< NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
< Server: cloudflare
< CF-RAY: 8095b304dccf39f0-YYZ
< alt-svc: h3=":443"; ma=86400
< 
* Connection #0 to host overseerr.greghao.com left intact

They reach cloudflare, but cloudflare can’t reach your server. Make sure port 80 is open and forwarded. Make sure your ISP isn’t blocking that port.

Doh!! So, my ISP does block 80 but not 443. This leads me back to the question above, can I run Caddy without having it pull the cert and instead use the Cloudflare SSL?

Probably. But that means you can only reach Caddy through Cloudflare, and requests coming from inside your network going directly to Caddy would not see a trusted certificate.

You could use the ACME DNS challenge (using the GitHub - caddy-dns/cloudflare: Caddy module: dns.providers.cloudflare plugin), or turn off Cloudflare proxying which would allow the ACME TLS-ALPN challenge to work (issuers connecting to port 443, directly reaching Caddy).

1 Like

As recommended, I’ve turned off Cloudflare proxy and certs have been renewed no problems. Thanks!

1 Like