Lets encrypt certificate renewal fails

Rob789 · May 5, 2021, 8:52pm

1. Caddy version (`caddy version`):

2.4.0-rc.1

2. How I run Caddy:

Native

a. System environment:

Debian Buster

b. Command:

Caddy start

d. My complete Caddyfile or JSON config:

# Global Option Block
{
        # General Option
        debug
}

# ACME Server
acme.roadrunner{
        acme_server
        tls internal
}

### REVERSE PROXY

## E-Mail
mail.mydomain.com {
        reverse_proxy https://192.168.2.1 {
                transport http {
                        tls_insecure_skip_verify
                }
        }
}

## Nextcloud
nextcloud.mydomain.com {
        reverse_proxy https://nextcloud.roadrunner{
                header_up Host {http.reverse_proxy.upstream.hostport}
                header_up X-Forwarded-Host {host}
        }
}

## Bitwarden
bitwarden.mydomain.com {
        reverse_proxy https://bitwarden.roadrunner{
                header_up Host {http.reverse_proxy.upstream.hostport}
                header_up X-Forwarded-Host {host}
        }
        respond /admin* "The admin panel is disabled, please configure the 'ADMIN_TOKEN' variable to enable it"
}

3. The problem I’m having:

At one point I could not connect to any services from outside the network. I discovered that my external IP address was changed so I updated the DNS. While waiting for the changes to become effective, I updated Caddy from 2.3.0 to 2.4.0-rc.1. Once completed I received the following errors when starting Caddy.

4. Error messages and/or full log output:

After caddy start

2021/05/05 20:43:04.718 INFO using adjacent Caddyfile
2021/05/05 20:43:04.725 WARN input is not formatted with ‘caddy fmt’ {“adapter”: “caddyfile”, “file”: “Caddyfile”, “line”: 17}
2021/05/05 20:43:04.730 INFO admin admin endpoint started {“address”: “tcp/localhost:2019”, “enforce_origin”: false, “origins”: [“localhost:2019”, “[::1]:2019”, “127.0.0.1:2019”]}
2021/05/05 20:43:04.743 INFO tls.cache.maintenance started background certificate maintenance {“cache”: “0xc00047ebd0”}
2021/05/05 20:43:04.762 INFO http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {“server_name”: “srv0”, “https_port”: 443}
2021/05/05 20:43:04.764 INFO http enabling automatic HTTP->HTTPS redirects {“server_name”: “srv0”}
2021/05/05 20:43:04.858 INFO pki.ca.local root certificate is already trusted by system {“path”: “storage:pki/authorities/local/root.crt”}
2021/05/05 20:43:04.862 DEBUG http starting server loop {“address”: “[::]:443”, “http3”: false, “tls”: true}
2021/05/05 20:43:04.862 DEBUG http starting server loop {“address”: “[::]:80”, “http3”: false, “tls”: false}
2021/05/05 20:43:04.863 INFO http enabling automatic TLS certificate management {“domains”: [“acme.roadrunner”, “nextcloud.mydomain.com”, “bitwarden.mydomain.com”, “mail.mydomain.com”]}
2021/05/05 20:43:04.865 WARN tls stapling OCSP {“error”: “no OCSP stapling for [acme.roadrunner]: no OCSP server specified in certificate”}
2021/05/05 20:43:04.870 INFO tls.obtain acquiring lock {“identifier”: “nextcloud.mydomain.com”}
2021/05/05 20:43:04.862 INFO tls cleaning storage unit {“description”: “FileStorage:/root/.local/share/caddy”}
2021/05/05 20:43:04.878 INFO tls finished cleaning storage units
2021/05/05 20:43:04.876 INFO tls.obtain acquiring lock {“identifier”: “mail.mydomain.com”}
2021/05/05 20:43:04.881 INFO autosaved config (load with --resume flag) {“file”: “/root/.config/caddy/autosave.json”}
2021/05/05 20:43:04.890 INFO serving initial configuration
2021/05/05 20:43:04.882 INFO tls.obtain acquiring lock {“identifier”: “bitwarden.mydomain.com”}
2021/05/05 20:43:04.895 INFO tls.obtain lock acquired {“identifier”: “bitwarden.mydomain.com”}
2021/05/05 20:43:04.882 INFO tls.obtain lock acquired {“identifier”: “nextcloud.mydomain.com”}
2021/05/05 20:43:04.900 INFO tls.issuance.acme waiting on internal rate limiter {“identifiers”: [“nextcloud.mydomain.com”]}
2021/05/05 20:43:04.902 INFO tls.issuance.acme done waiting on internal rate limiter {“identifiers”: [“nextcloud.mydomain.com”]}
2021/05/05 20:43:04.882 INFO tls.obtain lock acquired {“identifier”: “mail.mydomain.com”}
2021/05/05 20:43:04.907 INFO tls.issuance.acme waiting on internal rate limiter {“identifiers”: [“mail.mydomain.com”]}
2021/05/05 20:43:04.911 INFO tls.issuance.acme waiting on internal rate limiter {“identifiers”: [“bitwarden.mydomain.com”]}
2021/05/05 20:43:04.913 INFO tls.issuance.acme done waiting on internal rate limiter {“identifiers”: [“mail.mydomain.com”]}
2021/05/05 20:43:04.914 INFO tls.issuance.acme done waiting on internal rate limiter {“identifiers”: [“bitwarden.mydomain.com”]}
Successfully started Caddy (pid=2773) - Caddy is running in the background
root@RJ-CaddyTK ~# 2021/05/05 20:43:05.605 DEBUG tls.issuance.acme.acme_client http request {“method”: “GET”, “url”: “https://acme-v02.api.letsencrypt.org/directory”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Cache-Control”:[“public, max-age=0, no-cache”],“Content-Length”:[“658”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:05 GMT”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=604800”],“X-Frame-Options”:[“DENY”]}}
2021/05/05 20:43:05.778 DEBUG tls.issuance.acme.acme_client http request {“method”: “HEAD”, “url”: “https://acme-v02.api.letsencrypt.org/acme/new-nonce”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Cache-Control”:[“public, max-age=0, no-cache”],“Date”:[“Wed, 05 May 2021 20:43:05 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“0003ePKUZsDGvoAT-E5yv6xMXuvCh4njOJj946LNfW9AEmw”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=604800”],“X-Frame-Options”:[“DENY”]}}
2021/05/05 20:43:05.975 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme-v02.api.letsencrypt.org/acme/new-order”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 429, “response_headers”: {“Boulder-Requester”:[“122431579”],“Cache-Control”:[“public, max-age=0, no-cache”],“Content-Length”:[“201”],“Content-Type”:[“application/problem+json”],“Date”:[“Wed, 05 May 2021 20:43:05 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“0003wT7LOLukzuxa-DqNd4trDcAbgqBFbFwTTQy-ylgP50o”],“Server”:[“nginx”]}}
2021/05/05 20:43:05.983 WARN tls.issuance.zerossl missing email address for ZeroSSL; it is strongly recommended to set one for next time
2021/05/05 20:43:06.290 DEBUG tls.issuance.acme.acme_client http request {“method”: “HEAD”, “url”: “https://acme-v02.api.letsencrypt.org/acme/new-nonce”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Cache-Control”:[“public, max-age=0, no-cache”],“Date”:[“Wed, 05 May 2021 20:43:06 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“0004BYJOQPCTJ_nHtkIV8hW8e5SGXqWAz_RwCgVZuJ8LoD4”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=604800”],“X-Frame-Options”:[“DENY”]}}
2021/05/05 20:43:06.298 DEBUG tls.issuance.acme.acme_client http request {“method”: “HEAD”, “url”: “https://acme-v02.api.letsencrypt.org/acme/new-nonce”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Cache-Control”:[“public, max-age=0, no-cache”],“Date”:[“Wed, 05 May 2021 20:43:06 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“0003NH8oP_8EpMmbWK_0oOnKuuGF4OlhRuVHvL7BfSozZQQ”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=604800”],“X-Frame-Options”:[“DENY”]}}
2021/05/05 20:43:06.489 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme-v02.api.letsencrypt.org/acme/new-order”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 429, “response_headers”: {“Boulder-Requester”:[“122431579”],“Cache-Control”:[“public, max-age=0, no-cache”],“Content-Length”:[“201”],“Content-Type”:[“application/problem+json”],“Date”:[“Wed, 05 May 2021 20:43:06 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“0004ORARcSg188VuRIzTVT4q1KOKloqLTGEtYim7D4pbdt0”],“Server”:[“nginx”]}}
2021/05/05 20:43:06.493 WARN tls.issuance.zerossl missing email address for ZeroSSL; it is strongly recommended to set one for next time
2021/05/05 20:43:06.529 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme-v02.api.letsencrypt.org/acme/new-order”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 429, “response_headers”: {“Boulder-Requester”:[“122431579”],“Cache-Control”:[“public, max-age=0, no-cache”],“Content-Length”:[“201”],“Content-Type”:[“application/problem+json”],“Date”:[“Wed, 05 May 2021 20:43:06 GMT”],“Link”:[“https://acme-v02.api.letsencrypt.org/directory;rel="index"”],“Replay-Nonce”:[“0004pXratJ6PiXZ0C5tDRJyOEt5PH3hAY3Jg32PAcaZmbMo”],“Server”:[“nginx”]}}
2021/05/05 20:43:06.529 WARN tls.issuance.zerossl missing email address for ZeroSSL; it is strongly recommended to set one for next time
2021/05/05 20:43:06.596 INFO tls.issuance.zerossl generated EAB credentials {“key_id”: “1YJ-QywDLPNpG1LPBT7fWQ”}
2021/05/05 20:43:06.725 INFO tls.issuance.zerossl generated EAB credentials {“key_id”: “73b0SwA3lOYCctEOD0Ux2A”}
2021/05/05 20:43:06.764 INFO tls.issuance.zerossl generated EAB credentials {“key_id”: “BG8wiLbrzNvjRM6lmkOqSQ”}
2021/05/05 20:43:07.133 DEBUG tls.issuance.acme.acme_client http request {“method”: “GET”, “url”: “https://acme.zerossl.com/v2/DV90”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“645”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:07 GMT”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:07.510 DEBUG tls.issuance.acme.acme_client http request {“method”: “HEAD”, “url”: “https://acme.zerossl.com/v2/DV90/newNonce”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=-1”],“Content-Type”:[“application/octet-stream”],“Date”:[“Wed, 05 May 2021 20:43:07 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“hdH696klhglKit4WiYKERNMrvryKebM-c7OuYCkKmZ4”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:07.681 DEBUG tls.issuance.acme.acme_client http request {“method”: “HEAD”, “url”: “https://acme.zerossl.com/v2/DV90/newNonce”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=-1”],“Content-Type”:[“application/octet-stream”],“Date”:[“Wed, 05 May 2021 20:43:07 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“pZz6MfwKpd0LHg0shaxt-I5-n3jCMCokmRHLxOrEPpM”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:07.682 DEBUG tls.issuance.acme.acme_client http request {“method”: “HEAD”, “url”: “https://acme.zerossl.com/v2/DV90/newNonce”, “headers”: {“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=-1”],“Content-Type”:[“application/octet-stream”],“Date”:[“Wed, 05 May 2021 20:43:07 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“ZZatxb8BXdtuEx0h6J0Bwdv2H2C2dpQTUEXu6FybtAU”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:07.993 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/newAccount”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 201, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=0, no-cache, no-store”,“max-age=-1”],“Content-Length”:[“579”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:07 GMT”],“Location”:[“https://acme.zerossl.com/v2/DV90/account/1YJ-QywDLPNpG1LPBT7fWQ"],“Replay-Nonce”:[“aeVhcbnt5TKkwtnfq3ur4p887_ZArLeDtzAB1bfJXoc”],“Server”:[“nginx”],“Status”:[“”],“Strict-Transport-Security”:["max-age=15552000”]}}
2021/05/05 20:43:07.994 INFO tls.issuance.acme waiting on internal rate limiter {“identifiers”: [“nextcloud.mydomain.com”]}
2021/05/05 20:43:07.994 INFO tls.issuance.acme done waiting on internal rate limiter {“identifiers”: [“nextcloud.mydomain.com”]}
2021/05/05 20:43:08.201 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/newAccount”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 201, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=0, no-cache, no-store”,“max-age=-1”],“Content-Length”:[“579”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:08 GMT”],“Location”:[“https://acme.zerossl.com/v2/DV90/account/BG8wiLbrzNvjRM6lmkOqSQ"],“Replay-Nonce”:[“m-74eJpAJ_PS4Gh2ouyBQ6nOk0256djpfcx25Q1GDX0”],“Server”:[“nginx”],“Status”:[“”],“Strict-Transport-Security”:["max-age=15552000”]}}
2021/05/05 20:43:08.202 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/newAccount”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 201, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=0, no-cache, no-store”,“max-age=-1”],“Content-Length”:[“579”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:08 GMT”],“Location”:[“https://acme.zerossl.com/v2/DV90/account/73b0SwA3lOYCctEOD0Ux2A"],“Replay-Nonce”:[“9y-ZAfd-Xwgwgm0SwbPBFIW9oUOZT1XK0TRmr1035ps”],“Server”:[“nginx”],“Status”:[“”],“Strict-Transport-Security”:["max-age=15552000”]}}
2021/05/05 20:43:08.205 INFO tls.issuance.acme waiting on internal rate limiter {“identifiers”: [“bitwarden.mydomain.com”]}
2021/05/05 20:43:08.205 INFO tls.issuance.acme done waiting on internal rate limiter {“identifiers”: [“bitwarden.mydomain.com”]}
2021/05/05 20:43:08.207 INFO tls.issuance.acme waiting on internal rate limiter {“identifiers”: [“mail.mydomain.com”]}
2021/05/05 20:43:08.208 INFO tls.issuance.acme done waiting on internal rate limiter {“identifiers”: [“mail.mydomain.com”]}
2021/05/05 20:43:08.428 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/newOrder”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 201, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=0, no-cache, no-store”,“max-age=-1”],“Content-Length”:[“279”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:08 GMT”],“Location”:[“https://acme.zerossl.com/v2/DV90/order/Y7cN_qQEwoag53XUtupWoQ"],“Replay-Nonce”:[“OCbG-0Ayidd2fx7I6CwX2B4g5_ceatUbxw8Dz7_UPNM”],“Server”:[“nginx”],“Status”:[“”],“Strict-Transport-Security”:["max-age=15552000”]}}
2021/05/05 20:43:08.660 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/newOrder”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 201, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=0, no-cache, no-store”,“max-age=-1”],“Content-Length”:[“275”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:08 GMT”],“Location”:[“https://acme.zerossl.com/v2/DV90/order/X9S6V9TkZXy47S7wCfNINQ"],“Replay-Nonce”:[“8yLL4IYoE2l-klImkYZCHyrZIaQJNK0DMGUqscTPSY4”],“Server”:[“nginx”],“Status”:[“”],“Strict-Transport-Security”:["max-age=15552000”]}}
2021/05/05 20:43:08.680 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/newOrder”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 201, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=0, no-cache, no-store”,“max-age=-1”],“Content-Length”:[“274”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:08 GMT”],“Location”:[“https://acme.zerossl.com/v2/DV90/order/oIrSiQGuGMmnOuvZ8li_PQ"],“Replay-Nonce”:[“44TvnFBYmGkWjt1kTfbxD7OPkZIlcQLR2embOkaytN4”],“Server”:[“nginx”],“Status”:[“”],“Strict-Transport-Security”:["max-age=15552000”]}}
2021/05/05 20:43:08.842 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/authz/MPlGiDwaBBTJAwIfrUgphA”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“447”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:08 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[”_n-GQS_wMn-Z-kRno8a6EI_Sy86-DDK6cCKYt5BOVKs"],“Retry-After”:[“5”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:08.844 INFO tls.issuance.acme.acme_client trying to solve challenge {“identifier”: “nextcloud.mydomain.com”, “challenge_type”: “http-01”, “ca”: “https://acme.zerossl.com/v2/DV90”}
2021/05/05 20:43:09.075 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/authz/ozZ-c4G6dWZNV7qfuq7Dqw”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:["”],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“443”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:09 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“vl18xPp96gjpn4gVijjqy4VSaKlPkNkYCb4XRdPdB6s”],“Retry-After”:[“5”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:09.076 INFO tls.issuance.acme.acme_client trying to solve challenge {“identifier”: “bitwarden.mydomain.com”, “challenge_type”: “http-01”, “ca”: “https://acme.zerossl.com/v2/DV90”}
2021/05/05 20:43:09.095 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/authz/_vHl7VANaeYoE7ElvGa4Hw”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“442”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:09 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“xCUX8yINVGh4IZM0Ws6iOPDBesUHirzdShi6Cu1x2Uk”],“Retry-After”:[“5”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:09.097 INFO tls.issuance.acme.acme_client trying to solve challenge {“identifier”: “mail.mydomain.com”, “challenge_type”: “http-01”, “ca”: “https://acme.zerossl.com/v2/DV90”}
2021/05/05 20:43:09.266 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/chall/cUcZDAL5R2DgPr_FixNGAg”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“164”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:09 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”,“https://acme.zerossl.com/v2/DV90/authz/MPlGiDwaBBTJAwIfrUgphA;rel="up"”],“Replay-Nonce”:[“W3GoVKFZwhffpAG3T7Ft7evn8AQa1WFh-2bxNxAByGI”],“Retry-After”:[“10”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:09.268 DEBUG tls.issuance.acme.acme_client challenge accepted {“identifier”: “nextcloud.mydomain.com”, “challenge_type”: “http-01”}
2021/05/05 20:43:09.491 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/chall/4UJDr3WqjDmw1I4jpsk9pw”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“164”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:09 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”,“https://acme.zerossl.com/v2/DV90/authz/ozZ-c4G6dWZNV7qfuq7Dqw;rel="up"”],“Replay-Nonce”:[“O0meOoNxuvLicBfP_mWds2BXXd85s-_bX0xvb-fJqzc”],“Retry-After”:[“10”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:09.491 DEBUG tls.issuance.acme.acme_client challenge accepted {“identifier”: “bitwarden.mydomain.com”, “challenge_type”: “http-01”}
2021/05/05 20:43:09.504 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/chall/9AylfukRt2e1o4DS9SFuEA”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“164”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:09 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”,“https://acme.zerossl.com/v2/DV90/authz/_vHl7VANaeYoE7ElvGa4Hw;rel="up"”],“Replay-Nonce”:[“NZkCZPfGWKD0J7SHFz-Qm1g0V3DEjy0lWEFpyrNiz-E”],“Retry-After”:[“10”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:09.505 DEBUG tls.issuance.acme.acme_client challenge accepted {“identifier”: “mail.mydomain.com”, “challenge_type”: “http-01”}
2021/05/05 20:43:09.934 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/authz/MPlGiDwaBBTJAwIfrUgphA”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[““],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“450”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:09 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“pcyhPz4VqWOu8mAg1mc_j2HQ8XMQs8oKci6KvNXEvSw”],“Retry-After”:[“5”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:10.152 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/authz/ozZ-c4G6dWZNV7qfuq7Dqw”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[””],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“446”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:10 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“D_hLWrGhvaCxxTt5UNn-aKNbVbbHnu0IjIqnpuTSl30”],“Retry-After”:[“5”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}
2021/05/05 20:43:10.169 DEBUG tls.issuance.acme.acme_client http request {“method”: “POST”, “url”: “https://acme.zerossl.com/v2/DV90/authz/_vHl7VANaeYoE7ElvGa4Hw”, “headers”: {“Content-Type”:[“application/jose+json”],“User-Agent”:[“Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)”]}, “status_code”: 200, “response_headers”: {“Access-Control-Allow-Origin”:[“*”],“Cache-Control”:[“max-age=-1”],“Content-Length”:[“445”],“Content-Type”:[“application/json”],“Date”:[“Wed, 05 May 2021 20:43:10 GMT”],“Link”:[“https://acme.zerossl.com/v2/DV90;rel="index"”],“Replay-Nonce”:[“aFOG15AobVJI969asXY0MUj4s7RPviP3ftRf4Z3b09Y”],“Retry-After”:[“5”],“Server”:[“nginx”],“Strict-Transport-Security”:[“max-age=15552000”]}}

When I do caddy stop the log finished with:

2021/05/05 20:43:14.283 INFO admin.api received request {“method”: “POST”, “host”: “localhost:2019”, “uri”: “/stop”, “remote_addr”: “127.0.0.1:50846”, “headers”: {“Accept-Encoding”:[“gzip”],“Content-Length”:[“0”],“Origin”:[“localhost:2019”],“User-Agent”:[“Go-http-client/1.1”]}}
2021/05/05 20:43:14.283 WARN admin.api exiting; byeee!!
2021/05/05 20:43:14.286 INFO tls.cache.maintenance stopped background certificate maintenance {“cache”: “0xc00047ebd0”}
2021/05/05 20:43:14.286 DEBUG http.handlers.acme_server unloading unused CA database {“db_key”: “local”}
2021/05/05 20:43:14.290 WARN tls.issuance.acme.acme_client HTTP request failed; retrying {“url”: “https://acme.zerossl.com/v2/DV90/authz/ozZ-c4G6dWZNV7qfuq7Dqw”, “error”: “performing request: Post "https://acme.zerossl.com/v2/DV90/authz/ozZ-c4G6dWZNV7qfuq7Dqw\”: context canceled"}
2021/05/05 20:43:14.290 ERROR tls.issuance.acme.acme_client deactivating authorization {“identifier”: “bitwarden.mydomain.com”, “authz”: “https://acme.zerossl.com/v2/DV90/authz/ozZ-c4G6dWZNV7qfuq7Dqw”, “error”: “request to https://acme.zerossl.com/v2/DV90/authz/ozZ-c4G6dWZNV7qfuq7Dqw failed after 1 attempts: context canceled”}
2021/05/05 20:43:14.290 INFO tls.obtain releasing lock {“identifier”: “bitwarden.mydomain.com”}
2021/05/05 20:43:14.290 ERROR tls.obtain unable to unlock {“identifier”: “bitwarden.mydomain.com”, “lock_key”: “issue_cert_bitwarden.mydomain.com”, “error”: “remove /root/.local/share/caddy/locks/issue_cert_bitwarden.mydomain.com.lock: no such file or directory”}
2021/05/05 20:43:14.290 ERROR tls job failed {“error”: “bitwarden.mydomain.com: obtaining certificate: [bitwarden.mydomain.com] Obtain: [bitwarden.mydomain.com] solving challenges: [bitwarden.mydomain.com] context canceled (order=https://acme.zerossl.com/v2/DV90/order/X9S6V9TkZXy47S7wCfNINQ) (ca=https://acme.zerossl.com/v2/DV90)”}
2021/05/05 20:43:14.291 WARN tls.issuance.acme.acme_client HTTP request failed; retrying {“url”: “https://acme.zerossl.com/v2/DV90/authz/MPlGiDwaBBTJAwIfrUgphA”, “error”: “performing request: Post "https://acme.zerossl.com/v2/DV90/authz/MPlGiDwaBBTJAwIfrUgphA\”: context canceled"}
2021/05/05 20:43:14.293 ERROR tls.issuance.acme.acme_client deactivating authorization {“identifier”: “nextcloud.mydomain.com”, “authz”: “https://acme.zerossl.com/v2/DV90/authz/MPlGiDwaBBTJAwIfrUgphA”, “error”: “request to https://acme.zerossl.com/v2/DV90/authz/MPlGiDwaBBTJAwIfrUgphA failed after 1 attempts: context canceled”}
2021/05/05 20:43:14.293 INFO tls.obtain releasing lock {“identifier”: “nextcloud.mydomain.com”}
2021/05/05 20:43:14.293 INFO admin stopped previous server {“address”: “tcp/localhost:2019”}
2021/05/05 20:43:14.294 INFO admin.api shutdown complete {“exit_code”: 0}

5. What I already tried:

I removed the certificates and db manually without result.

6. Links to relevant resources:

francislavoie · May 5, 2021, 11:28pm

You’re missing a space here before the {. Spaces are important in the Caddyfile!

You killed Caddy before it was done trying to do issuance (see “context cancelled”, this is Caddy saying “I was told to cancel all the work I was doing cause I was told to shut down”)

Just let it run a bit longer, it was still working on it.

Also, consider running Caddy as a systemd service instead of using caddy start, it’s more reliable that way (if Caddy dies, it will be automatically restarted, etc).

Also I recommend using ``` for posting logs on the forums, it makes it much easier to read (text is aligned, monospaced font, scrollable box, etc)

Rob789 · May 6, 2021, 4:28pm

Sorry that was a C&P error.

I should have been more clear by mentioning that I stopped Caddy after a while. But to be sure I ran Caddy again and let it try to get a certificate for 20 minutes or so. Still same result, just longer log with errors.

I do (or I should say did) under normal conditions but I find it easier to read the debug log live. Anyway, for the sake of testing, I moved back to systemd and below is the journalcltl


May 06 16:08:01 RJ-CaddyTK systemd[1]: Starting Caddy...
May 06 16:08:01 RJ-CaddyTK caddy[4226]: caddy.HomeDir=/var/lib/caddy
May 06 16:08:01 RJ-CaddyTK caddy[4226]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
May 06 16:08:01 RJ-CaddyTK caddy[4226]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
May 06 16:08:01 RJ-CaddyTK caddy[4226]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
May 06 16:08:01 RJ-CaddyTK caddy[4226]: caddy.Version=v2.4.0-rc.1 h1:tZl6bDhlwtRwuWpebRUYpDJPhJaGyrXIMp7fmuMXwMc=
May 06 16:08:01 RJ-CaddyTK caddy[4226]: runtime.GOOS=linux
May 06 16:08:01 RJ-CaddyTK caddy[4226]: runtime.GOARCH=amd64
May 06 16:08:01 RJ-CaddyTK caddy[4226]: runtime.Compiler=gc
May 06 16:08:01 RJ-CaddyTK caddy[4226]: runtime.NumCPU=1
May 06 16:08:01 RJ-CaddyTK caddy[4226]: runtime.GOMAXPROCS=1
May 06 16:08:01 RJ-CaddyTK caddy[4226]: runtime.Version=go1.16.3
May 06 16:08:01 RJ-CaddyTK caddy[4226]: os.Getwd=/
May 06 16:08:01 RJ-CaddyTK caddy[4226]: LANG=en_US.UTF-8
May 06 16:08:01 RJ-CaddyTK caddy[4226]: LANGUAGE=en_US.UTF-8
May 06 16:08:01 RJ-CaddyTK caddy[4226]: LC_CTYPE=C
May 06 16:08:01 RJ-CaddyTK caddy[4226]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
May 06 16:08:01 RJ-CaddyTK caddy[4226]: NOTIFY_SOCKET=/run/systemd/notify
May 06 16:08:01 RJ-CaddyTK caddy[4226]: HOME=/var/lib/caddy
May 06 16:08:01 RJ-CaddyTK caddy[4226]: LOGNAME=caddy
May 06 16:08:01 RJ-CaddyTK caddy[4226]: USER=caddy
May 06 16:08:01 RJ-CaddyTK caddy[4226]: INVOCATION_ID=c0d8bf76361d454dbb45b1fedcd48607
May 06 16:08:01 RJ-CaddyTK caddy[4226]: JOURNAL_STREAM=9:216942
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.4378924,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"warn","ts":1620317281.453002,"msg":"input is not formatted with 'caddy fmt'","adapter":"caddyfile","file":"/etc/caddy/Caddyfile","line":17}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.4579294,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["127.0.0.1:2019","localhost:2019","[::1]:2019"]}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.468751,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002551f0"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.4921334,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.4925091,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"warn","ts":1620317281.5968354,"logger":"pki.ca.local","msg":"installing root certificate (you might be prompted for password)","path":"storage:pki/authorities/local/root.crt"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: 2021/05/06 16:08:01 Warning: "certutil" is not available, install "certutil" with "apt install libnss3-tools" or "yum install nss-tools" and try again
May 06 16:08:01 RJ-CaddyTK caddy[4226]: 2021/05/06 16:08:01 define JAVA_HOME environment variable to use the Java trust
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"error","ts":1620317281.613561,"logger":"pki.ca.local","msg":"failed to install root certificate","error":"failed to execute tee: exit status 1","certificate_file":"storage:pki/authorities/local/root.crt"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317281.615104,"logger":"http","msg":"starting server loop","address":"[::]:443","http3":false,"tls":true}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317281.6156528,"logger":"http","msg":"starting server loop","address":"[::]:80","http3":false,"tls":false}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6163545,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["nextcloud.mydomain.com","bitwarden.mydomain.com","mail.mydomain.com","acme.roadrunner"]}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6176584,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
May 06 16:08:01 RJ-CaddyTK systemd[1]: Started Caddy.
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6186373,"logger":"tls.obtain","msg":"acquiring lock","identifier":"acme.roadrunner"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.616273,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6275635,"logger":"tls","msg":"finished cleaning storage units"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.620026,"logger":"tls.obtain","msg":"acquiring lock","identifier":"bitwarden.mydomain.com"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6205237,"logger":"tls.obtain","msg":"acquiring lock","identifier":"mail.mydomain.com"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6274009,"logger":"tls.obtain","msg":"lock acquired","identifier":"acme.roadrunner"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6339977,"logger":"tls.obtain","msg":"certificate obtained successfully","identifier":"acme.roadrunner"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6345909,"logger":"tls.obtain","msg":"releasing lock","identifier":"acme.roadrunner"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"warn","ts":1620317281.6368713,"logger":"tls","msg":"stapling OCSP","error":"no OCSP stapling for [acme.roadrunner]: no OCSP server specified in certificate"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6345098,"logger":"tls.obtain","msg":"lock acquired","identifier":"mail.mydomain.com"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6345692,"logger":"tls.obtain","msg":"lock acquired","identifier":"bitwarden.mydomain.com"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.619443,"logger":"tls.obtain","msg":"acquiring lock","identifier":"nextcloud.mydomain.com"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6286778,"msg":"serving initial configuration"}
May 06 16:08:01 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317281.6441624,"logger":"tls.obtain","msg":"lock acquired","identifier":"nextcloud.mydomain.com"}
May 06 16:08:02 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317282.55228,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","headers":{"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["658"],"Content-Type":["
application/json"],"Date":["Thu, 06 May 2021 16:08:02 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:02 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317282.7146304,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 06 May 2021 16:08:02
 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["01038At_jIZaJehUT-63V8jZNlX5tVu3L4497EsmeoZzny4"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:02 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317282.9790657,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["122544650"],"Cache-C
ontrol":["public, max-age=0, no-cache"],"Content-Length":["286"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:02 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/122544650"],"Replay-Nonce":["0103rGIchqS_pCzK
KqVLBTU8UBygvSg7rJR82vTfezoYEl0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:02 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317282.9814744,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["mail.mydomain.com"]}
May 06 16:08:02 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317282.9819522,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["mail.mydomain.com"]}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.205624,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 06 May 2021 16:08:03
GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0104nQce0g9gTR2ZaQt9hExh_bye8pbc1Vdv5Ga-W-VzeUM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.250786,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","headers":{"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Thu, 06 May 2021 16:08:03
GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0004ovf6pcnD81xNaIM-oky1sd_KcS6d-UBO9rGLdAKMCgg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.3730311,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["122544650"],"Cache-
Control":["public, max-age=0, no-cache"],"Content-Length":["334"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/122544650/9523040999"],"Replay-Nonce":["0103ScHiUWXzqX-M2UxofSBNVScnotApm7VtbuIBjUl4WuA"],"Server":["nginx"],"Strict-Transport-Security":[
"max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.516409,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["122544652"],"Cache-Co
ntrol":["public, max-age=0, no-cache"],"Content-Length":["286"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/122544652"],"Replay-Nonce":["0104sTRfp5r6w6Zii
X3TEn5RoM_-wsX6x-WuJTGDpd5758Y"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317283.5182803,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["nextcloud.mydomain.com"]}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317283.518779,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["nextcloud.mydomain.com"]}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.5450892,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-acct","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["122544653"],"Cache-C
ontrol":["public, max-age=0, no-cache"],"Content-Length":["286"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf>;rel=\"terms-of-service\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/acct/122544653"],"Replay-Nonce":["00045d6cIVa6Px10
uVoP_i0QzL8V_LLXPtSwh4JauGvqILg"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317283.5467453,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["bitwarden.mydomain.com"]}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317283.5473151,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["bitwarden.mydomain.com"]}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.5732002,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/12902165303","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["12254465
0"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["793"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0103OffznPTEaxSvTtsVsRft-eMp2yz3s2F6o7S7d7TU0UM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.5735729,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"dns-01"}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317283.5735943,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mail.mydomain.com","challenge_type":"http-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.7765467,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["122544653"],"Cache-
Control":["public, max-age=0, no-cache"],"Content-Length":["335"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/122544653/9523041443"],"Replay-Nonce":["0004EIaxBYNMvgaMfokxfxFi92LzejOQp3t8EhQZAmGBg9M"],"Server":["nginx"],"Strict-Transport-Security":[
"max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.7971518,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/chall-v3/12902165303/3P-jSw","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["1
22544650"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["186"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\"","<https://acme-v02.api.letsencrypt.org/acme/authz-v3/12902165303>;rel=\"up\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/chall-v3/12902165303/3P-jSw"],"Replay-Nonce":["0
103ByhoZi2elomajRvkKb_hiytJOr_ZVyT8fcHzunmntlk"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.7979572,"logger":"tls.issuance.acme.acme_client","msg":"challenge accepted","identifier":"mail.mydomain.com","challenge_type":"http-01"}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.857994,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":201,"response_headers":{"Boulder-Requester":["122544652"],"Cache-C
ontrol":["public, max-age=0, no-cache"],"Content-Length":["339"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.letsencrypt.org/acme/order/122544652/9523041516"],"Replay-Nonce":["0103vT3O_JNoPV8qKGbuWuoAclnqWmjoiUJs9ugJFFl4BQ0"],"Server":["nginx"],"Strict-Transport-Security":["
max-age=604800"],"X-Frame-Options":["DENY"]}}
May 06 16:08:03 RJ-CaddyTK caddy[4226]: {"level":"debug","ts":1620317283.9580688,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/12902165803","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.4.0-rc.1 CertMagic acmez (linux; amd64)"]},"status_code":200,"response_headers":{"Boulder-Requester":["12254465
3"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["794"],"Content-Type":["application/json"],"Date":["Thu, 06 May 2021 16:08:03 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["0003Yr0I4SxoSSp2WZTFWIR7qyXG5Xq2T6TW3Wnwic_0oCU"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}}

Edit: looking again more carefully to the log output, I see the following lines coming back after x attempts for each subdomain.

May 06 16:08:27 RJ-CaddyTK caddy[4226]: {"level":"error","ts":1620317307.0265257,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"mail.mydomain.com","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:connection","error":"Timeout during connect (likely firewall problem)"}
May 06 16:08:27 RJ-CaddyTK caddy[4226]: {"level":"error","ts":1620317307.0272741,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"mail.mydomain.com","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:connection - Timeout during connect (likely firewall problem)","order":"https://acme-v02.api.letsencrypt.org/acme/order/122544650/9523045462","attempt":2,"max_attempts":3}
May 06 16:08:32 RJ-CaddyTK caddy[4226]: {"level":"info","ts":1620317312.343098,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"mail.mydomain.com","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}

francislavoie · May 6, 2021, 4:45pm

Are you 100% certain that you don’t have something between your server and let’s encrypt? Like maybe CloudFlare which isn’t letting the request through? Are you sure your firewall/port forwarding is correct? Are you certain DNS records are correct?

Rob789 · May 6, 2021, 6:17pm

Well, the only thing I’m certain about is that computers take a lot of time! Everything else I’m never certain.

I did check the firewall logs before my first post and I checked the DNS. But I checked it al again plus I disabled the blacklist in the unbound DNS in my firewall.

Your last post gave me more certainty that it must be something to do with the connection so I went in my modem to look for an indication what’s going on and I think I found it.

As I mentioned, my external IP address was recently changed. I did notice that it was a complete different range but didn’t pay further attention to it. But in the modem I can see my IP address has been given a port range (Ports 16384 à 32767). I heard a while ago that my provider started sharing IP addresses among customers due to the lack of IPv4 addresses by giving the same IP address with different port ranges. Until the last IP change I was the lucky one with a dedicated IP. Now it looks like I’m not any more.

I double checked this by bypassing Caddy completely and forward port 443 directly to my email server as I had before.

I guess all I can do now is start shouting at my ISP

matt · May 6, 2021, 6:50pm

I wrote a Caddy plugin that will keep your DNS records updated with your dynamic IP:

francislavoie · May 6, 2021, 6:53pm

I just ran into a thread on reddit.com/r/selfhosting mentioning this tool:

It might be exactly what you need. Basically you’d buy a cheap VPS (like $5/m or something, whichever provider of your choosing) then set this up to join it into your home network via WireGuard, then you can point your DNS to the IP of your VPS instead and the traffic will be piped to your home network.

Rob789 · May 6, 2021, 7:52pm

That sounds cool! I currently run a separate script to update my DNS record (on a different host) but it would be so much better to have this all done by Caddy on the same host.

That’s looks like a pretty awesome tool. Also to cover some other issues i’m facing.

Now that I know the core reason of my problem, I’m first going to bug my ISP. Maybe they have an option to undo this change. For a “fair” charge of course…

Whitestrake · May 7, 2021, 1:27am

That seems like an interesting approach. Basically CG-NAT plus a big port forward range?

If it’s anything like CG-NAT, fingers crossed you’ll be able to opt-out without a problem.

Punching out from the home network to your VPS is probably the easier way to go rather than trying to get the VPS to find your home network. Means you can leave your home network fully firewalled, too, and the VPS isn’t going to be jumping around IP addresses or port ranges.

Would also throw “permanent reverse SSH tunnel” into the ring as a good solution instead of a full VPN if you’re going to go the VPS route.

francislavoie · May 7, 2021, 2:24am

I agree re “reverse SSH tunnel”, I spoke with @Whitestrake off-site and he reminded me how cool that feature is.

Basically looks like:

autossh -o "ExitOnForwardFailure=yes" \
    -fN -M 10984 \
    -R 80:localhost:80 \
    -R 443:localhost:443 \
    username@remoteserver &

Basically the idea is that on your home machine, you run this to open an SSH tunnel to the VPS remote server, and it’ll make all incoming connections to port 80 and 443 on the VPS get passed through the tunnel back to your home machine.

The tool autossh is just a wrapper around ssh which keeps the tunnel running persistently, restarting it if it goes down. Some additional options on there that I think are useful or necessary.

(Note that you can also set up tunnels in the other direction with -L which is more common usually, so that you can make connections to a specific port on your local machine get tunnelled to your remote machine, e.g. to talk to Caddy’s localhost:2019 admin endpoint if Caddy was on another machine, as if it was on your local machine).

Some additional reading:

https://raymii.org/s/tutorials/Autossh_persistent_tunnels.html

https://www.harding.motd.ca/autossh/README.txt

basil · May 7, 2021, 5:02am

This caught my attention. It’s not clear how to include the plugin in a build though. It doesn’t appear to be available from the download page.

This is something new and novel and somewhat disconcerting.

As if CG-NAT hasn’t complicated matters enough already and along comes a new twist. CG-NAT and reverse proxies are just bad juju. It’s like oil and water; they don’t mix.

While there are a lot more upsides than downsides with tunnelling technologies, on the downside, I believe there is some impact on upload/download internet speeds. As upload speeds are generally quite a bit lower than download speeds, residential type connections are especially sensitive and reverse proxies take a bit of a hit. Whether it’s noticeable or not, I’m not sure?

matt · May 7, 2021, 5:51am

Working on that; in the meantime, xcaddy makes this really easy, as described in our docs: Build from source — Caddy Documentation

Rob789 · May 7, 2021, 10:58am

I’ve come across this aaproach when reading up on mTLS (if I remember correctly on the smallstep website). It sounds like a really good alternative for VPN and especially for smartphones. Instead of keeping a battery draining vpn connection that also usually closes when in idle.

Rob789 · May 22, 2021, 8:23am

Just an update to let everyone know it is all working again. I was able to opt out the limited IP and have a full stack IP again. Once that was back in place, Caddy was working again as expected.

system · June 4, 2021, 8:52pm

This topic was automatically closed after 30 days. New replies are no longer allowed.