Certificate issuing problems with Caddy, Proxmox VE while ISP is blocking port :80

Help
1

Hi @emilylange ,

this was working well for me until I did a fresh ubuntu install and copied over my Caddyfile and docker-compose.

Now for some reason www.naff.casa isn’t able to obtain a cert, even though the other caddy’s are able to fine (except for proxmox which is my next question).

Any idea why proxmox isn’t able to obtain a cert? I had it working previously as well, but not working now:

{
    # Global options block. Entirely optional, https is on by default
    # Optional email key for lets encrypt
    email ryan.naff@gmail.com
    # Optional staging lets encrypt for testing. Comment out for production.
#     acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
port.naff.casa {
    reverse_proxy 192.168.1.37:9000
}
www.naff.casa {
    redir https://naff.casa{uri}
}
naff.casa {
    reverse_proxy 192.168.1.37:2368
}
sonarr.naff.casa {
    reverse_proxy 192.168.1.37:8989
}
plex.naff.casa {
    reverse_proxy 192.168.1.37:32400
}
radarr.naff.casa {
    reverse_proxy 192.168.1.37:7878
}
ombi.naff.casa {
    reverse_proxy 192.168.1.37:3579
}
remote.naff.casa {
    reverse_proxy 192.168.1.37:6969
}
bw.naff.casa {
    reverse_proxy 192.168.1.37:8711
}
cctv.naff.casa {
    reverse_proxy 192.168.1.39:8123
}
jellyfin.naff.casa {
    reverse_proxy 192.168.1.37:8096
}
code.naff.casa {
    reverse_proxy 192.168.1.37:4443
}
prox.naff.casa {
reverse_proxy 192.168.1.218:8006 {
transport http {
tls_insecure_skip_verify
}
}
}
2

Hi :slight_smile:

I moved your post into a new topic :innocent:

Would be cool if you could - for the sake of simplicity and the sheer length of that original topic - fill out the forum template again.

For that you have to click on “New Topic”, select the category “Help” and copy it from there into a new post here (or just open a new topic, and I’ll merge it with this one here)

That being said :slight_smile:

I am aware that Proxmox VE has some built-in function to issue a certificate for your node via ACME, but that’s not what you mean, is it?

Also, feel free to edit that topic title to whatever you think might be more fitting.

It’s - as of now - somewhat unclear to me what your two problems actually are :see_no_evil:

2 Likes
3

Hi @emilylange,

My problem is that prox.naff.casa doesn’t work.

I had it working a couple weeks ago but now after a new fresh install, for some reason it’s not working.

Also, my www.naff.casa isn’t working either, and you helped me with it month or so ago and it worked at the time.

4

Howdy @naffhouse,

Unfortunately, ‘doesn’t work’ is one of those phrases that’s the bane of people providing support everywhere. There’s a lot of possible failure modes and ‘doesn’t work’ doesn’t help us figure out what failed, only that something did; we don’t need to know that it has failed, we need to know how so we can figure out why and help you.

You’ve said that you migrated your Caddy web server and configuration over to a fresh Ubuntu installation, and that www.naff.casa can no longer obtain a certificate. Then you’ve asked why Proxmox can’t obtain a certificate. Is Proxmox running ACME or are you referring to Caddy on Ubuntu on Proxmox here? When you say “other caddy’s”, what do you mean, exactly - Caddy servers on other physical hosts? Caddy servers on other virtual hosts on the same machine? Caddy servers in VPSs? Help paint a clear picture for us so we can envision the environment you’re referring to.

You said your prox.naff.casa and www.naff.casa aren’t working now. Things don’t go from ‘working’ to ‘not working’ without something changing and logs will indicate this. You’ve posted a Caddyfile with those two sites alongside a number of other sites, but you haven’t specified whether those other sites are still functioning fine or not; I made a few connections to those sites and found they responded OK.

What’s happening in the logs when your Caddy server is trying to obtain these certificates?

3 Likes
5

just fix it.

hahah just kidding man.

Sorry for not providing enough info and not being detailed.

If you need me to post further up in the log history let me know and I’ll do a pastebin.

Here are my logs from my docker-compose container, let me know if there are other logs I can provide.

9f3369032ef7c142943"}
{"level":"info","ts":1663271733.1643457,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["www.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663271733.1644254,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["www.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"error","ts":1663271733.3689177,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.naff.casa","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"info","ts":1663271736.6541646,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/68566534/4088028534"}
{"level":"info","ts":1663271737.117922,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa08aa8520a929984ef68b9a17c089ba9079"}
{"level":"info","ts":1663271737.1188936,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["prox.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663271737.1189651,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["prox.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"error","ts":1663271737.214081,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"prox.naff.casa","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"info","ts":1663271744.877628,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"www.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1663271753.3840153,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"prox.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"error","ts":1663272058.3240888,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[www.naff.casa] solving challenges: [www.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/GuDHHBz6cz7-o0ZnhqV5kA) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663272058.3242047,"logger":"tls.obtain","msg":"will retry","error":"[www.naff.casa] Obtain: [www.naff.casa] solving challenges: [www.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/GuDHHBz6cz7-o0ZnhqV5kA) (ca=https://acme.zerossl.com/v2/DV90)","attempt":10,"retrying_in":3600,"elapsed":10552.865602505,"max_duration":2592000}
{"level":"error","ts":1663272071.7367034,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"prox.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[prox.naff.casa] solving challenges: [prox.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/EY7WqfjQ-p3dkWPinidbzg) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663272071.7368155,"logger":"tls.obtain","msg":"will retry","error":"[prox.naff.casa] Obtain: [prox.naff.casa] solving challenges: [prox.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/EY7WqfjQ-p3dkWPinidbzg) (ca=https://acme.zerossl.com/v2/DV90)","attempt":10,"retrying_in":3600,"elapsed":10566.276453127,"max_duration":2592000}
{"level":"info","ts":1663275658.653685,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/68566534/4088698554"}
{"level":"info","ts":1663275659.1501353,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa9a224f7b16a7084dfdf9dc7b4067eca812"}
{"level":"info","ts":1663275659.1507766,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["www.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663275659.1508584,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["www.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"error","ts":1663275659.3639143,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.naff.casa","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"info","ts":1663275671.8869538,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/68566534/4088701494"}
{"level":"info","ts":1663275672.3881118,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fae6a49b082eb282722ff7055cd347f6f158"}
{"level":"info","ts":1663275672.3886328,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["prox.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663275672.388678,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["prox.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"error","ts":1663275672.4913423,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"prox.naff.casa","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"info","ts":1663275678.1906085,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"www.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1663275684.3483717,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"prox.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"error","ts":1663275992.2135859,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[www.naff.casa] solving challenges: [www.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/n0hoy2CoQNkpnbjA8mLN1Q) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663275992.2136986,"logger":"tls.obtain","msg":"will retry","error":"[www.naff.casa] Obtain: [www.naff.casa] solving challenges: [www.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/n0hoy2CoQNkpnbjA8mLN1Q) (ca=https://acme.zerossl.com/v2/DV90)","attempt":11,"retrying_in":10800,"elapsed":14486.755103113,"max_duration":2592000}
{"level":"error","ts":1663276003.5016613,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"prox.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[prox.naff.casa] solving challenges: [prox.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/CEeBRWMoo6S3hwKF9K3GIw) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663276003.5017574,"logger":"tls.obtain","msg":"will retry","error":"[prox.naff.casa] Obtain: [prox.naff.casa] solving challenges: [prox.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/CEeBRWMoo6S3hwKF9K3GIw) (ca=https://acme.zerossl.com/v2/DV90)","attempt":11,"retrying_in":10800,"elapsed":14498.041393039,"max_duration":2592000}
{"level":"info","ts":1663286792.5617113,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/68566534/4090641874"}
{"level":"info","ts":1663286793.1029058,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fade9eef11a786d44b2057a05052d415037e"}
{"level":"info","ts":1663286793.1035993,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["www.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663286793.1036797,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["www.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"error","ts":1663286793.3147075,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"www.naff.casa","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"info","ts":1663286803.653952,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/68566534/4090643364"}
{"level":"info","ts":1663286804.072883,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa15badf57db7ae82d596c4f305cbb1e4bfb"}
{"level":"info","ts":1663286804.0735164,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["prox.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663286804.0735745,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["prox.naff.casa"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":"ryan.naff@gmail.com"}
{"level":"error","ts":1663286804.1730857,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"prox.naff.casa","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:malformed - JWS verification error"}
{"level":"info","ts":1663286816.1467965,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"www.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1663286823.1594443,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"prox.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
6

I’m seeing a lot of JWS verification errors. It looks like we have seen this before, actually:

There are indications that this occurs when the account ID and the key is mismatched: JWS verification error · Issue #1006 · go-acme/lego · GitHub

When you migrated to your new Ubuntu instance, did you copy your Caddy data folders with it? Regardless, I’d suggest nuking it - accounts, certificates, keys, the lot, just move them to some backup folder that Caddy can’t see - and then acquiring all new certificates.

1 Like
7

Hey Matt,

When I migrated I just copied over caddyfile and compose, nothing else.

To fully nuke, I delete everything in caddy folder and anything else?

To reset the keys, should I use a different email in my caddyfile?

This is my first time doing this.

Anything else I’m not covering?

8

Just this folder from your docker-compose.yml in your original post:

You can just bring the Caddy container down and then move the entire folder, like:

mv /docker/caddy/data{,.bak}

Or similar, to keep it on hand but force Caddy to generate everything else fresh.

Remove restart: unless-stopped for this run and then docker compose up caddy (with no -d flag) to run it in the foreground, capture the log output, and see if it looks good. If it’s good, CTRL+C out and then compose it up again with -d. If it’s bad, post the logs here.

1 Like
9

very bizzare, every single one seemed to get it’s cert except for Sonarr.

I thought maybe Sonarr was down so I connected to it via it’s internal IP, and was able to get to the web gui.

I thought maybe my Caddyfile may have Sonarr setup with the wrong ping, but it didn’t.

here's my log:

{"level":"error","ts":1663296053.3924172,"logger":"http.log.error","msg":"dial tcp 192.168.1.37:8814: connect: connection refused","request":{"remote_ip":"39.110.218.101","remote_port":"62178","proto":"HTTP/1.1","method":"GET","host":"cloud.naff.casa","uri":"/","headers":{"Accept-Language":["ja,en-US;q=0.9,en;q=0.8"],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"User-Agent":["Mozilla/5.0 (Linux; U; Android 2.2; ja-jp; SC-02B Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"cloud.naff.casa"}},"duration":0.00074489,"status":502,"err_id":"39b50kqdd","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1663296066.9390113,"logger":"http.log.error","msg":"dial tcp 192.168.1.37:32400: connect: connection refused","request":{"remote_ip":"39.110.218.101","remote_port":"62186","proto":"HTTP/1.1","method":"GET","host":"plex.naff.casa","uri":"/","headers":{"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"],"User-Agent":["Mozilla/5.0 (Linux; U; Android 2.2; ja-jp; SC-02B Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"],"Accept-Language":["ja,en-US;q=0.9,en;q=0.8"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"plex.naff.casa"}},"duration":0.000593006,"status":502,"err_id":"ukcukk9jr","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1663296082.2149818,"logger":"http.log.error","msg":"dial tcp 192.168.1.37:8814: connect: connection refused","request":{"remote_ip":"39.110.218.101","remote_port":"62192","proto":"HTTP/1.1","method":"GET","host":"cloud.naff.casa","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (Linux; U; Android 2.2; ja-jp; SC-02B Build/FROYO) AppleWebKit/533.1 (KHTML, like Gecko) Version/4.0 Mobile Safari/533.1"],"Accept-Language":["ja,en-US;q=0.9,en;q=0.8"],"Accept":["*/*"],"Accept-Encoding":["gzip, deflate"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"cloud.naff.casa"}},"duration":0.000640417,"status":502,"err_id":"4vr36c4hd","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1663296226.3120346,"logger":"http.log.error","msg":"dial tcp 192.168.1.37:8814: connect: connection refused","request":{"remote_ip":"179.43.169.181","remote_port":"35726","proto":"HTTP/1.1","method":"GET","host":"cloud.naff.casa","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"cloud.naff.casa"}},"duration":0.001017424,"status":502,"err_id":"8yjhk6pgn","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1663296229.0394917,"logger":"http.log.error","msg":"dial tcp 192.168.1.37:32400: connect: connection refused","request":{"remote_ip":"179.43.169.181","remote_port":"35768","proto":"HTTP/1.1","method":"GET","host":"plex.naff.casa","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"plex.naff.casa"}},"duration":0.000634264,"status":502,"err_id":"fjijzsxjv","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1663296231.8684978,"logger":"http.log.error","msg":"dial tcp 192.168.1.37:32400: connect: connection refused","request":{"remote_ip":"179.43.169.181","remote_port":"35768","proto":"HTTP/1.1","method":"GET","host":"plex.naff.casa","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"plex.naff.casa"}},"duration":0.000664287,"status":502,"err_id":"fs66zp67d","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1663296233.925612,"logger":"http.log.error","msg":"dial tcp 192.168.1.37:8814: connect: connection refused","request":{"remote_ip":"179.43.169.181","remote_port":"35726","proto":"HTTP/1.1","method":"GET","host":"cloud.naff.casa","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4867,"proto":"","server_name":"cloud.naff.casa"}},"duration":0.00064863,"status":502,"err_id":"v9ccpzfhs","err_trace":"reverseproxy.statusError (reverseproxy.go:1184)"}
{"level":"error","ts":1663296290.0770724,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"bw.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[bw.naff.casa] solving challenges: [bw.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/8Vp3y_9a3kRbZLQnKoes2Q) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663296290.0771823,"logger":"tls.obtain","msg":"will retry","error":"[bw.naff.casa] Obtain: [bw.naff.casa] solving challenges: [bw.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/8Vp3y_9a3kRbZLQnKoes2Q) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":332.219629785,"max_duration":2592000}
{"level":"info","ts":1663296290.357943,"msg":"shutting down apps, then terminating","signal":"SIGTERM"}
{"level":"warn","ts":1663296290.3580534,"msg":"exiting; byeee!! 👋","signal":"SIGTERM"}
{"level":"info","ts":1663296290.376374,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0004c68c0"}
{"level":"info","ts":1663296290.376639,"logger":"tls.obtain","msg":"releasing lock","identifier":"bw.naff.casa"}
{"level":"warn","ts":1663296290.3767073,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/authz/xNW-Y3sDCRFWMcP2tTApYw","error":"performing request: Post \"https://acme.zerossl.com/v2/DV90/authz/xNW-Y3sDCRFWMcP2tTApYw\": context canceled"}
{"level":"error","ts":1663296290.3767726,"logger":"tls.issuance.acme.acme_client","msg":"deactivating authorization","identifier":"cctv.naff.casa","authz":"https://acme.zerossl.com/v2/DV90/authz/xNW-Y3sDCRFWMcP2tTApYw","error":"attempt 1: https://acme.zerossl.com/v2/DV90/authz/xNW-Y3sDCRFWMcP2tTApYw: context canceled"}
{"level":"error","ts":1663296290.376856,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"cctv.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[cctv.naff.casa] solving challenges: [cctv.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/WNJPI_odkUd53fLzWmN9wQ) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663296290.3768992,"logger":"tls.obtain","msg":"will retry","error":"[cctv.naff.casa] Obtain: [cctv.naff.casa] solving challenges: [cctv.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/WNJPI_odkUd53fLzWmN9wQ) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":332.520456778,"max_duration":2592000}
{"level":"info","ts":1663296290.376948,"logger":"tls.obtain","msg":"releasing lock","identifier":"cctv.naff.casa"}
{"level":"error","ts":1663296290.3771071,"msg":"unable to clean up lock in storage backend","signal":"SIGTERM","storage":"FileStorage:/data/caddy","lock_key":"issue_cert_bw.naff.casa","error":"remove /data/caddy/locks/issue_cert_bw.naff.casa.lock: no such file or directory"}
{"level":"error","ts":1663296290.3771815,"msg":"unable to clean up lock in storage backend","signal":"SIGTERM","storage":"FileStorage:/data/caddy","lock_key":"issue_cert_cctv.naff.casa","error":"remove /data/caddy/locks/issue_cert_cctv.naff.casa.lock: no such file or directory"}
{"level":"warn","ts":1663296290.3771977,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/authz/vEdls6HtMUfGurgTQFL9Cg","error":"performing request: Post \"https://acme.zerossl.com/v2/DV90/authz/vEdls6HtMUfGurgTQFL9Cg\": context canceled"}
{"level":"error","ts":1663296290.3772585,"logger":"tls.issuance.acme.acme_client","msg":"deactivating authorization","identifier":"sonarr.naff.casa","authz":"https://acme.zerossl.com/v2/DV90/authz/vEdls6HtMUfGurgTQFL9Cg","error":"attempt 1: https://acme.zerossl.com/v2/DV90/authz/vEdls6HtMUfGurgTQFL9Cg: context canceled"}
{"level":"error","ts":1663296290.3773367,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sonarr.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[sonarr.naff.casa] solving challenges: [sonarr.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/PRSIphpSLbOmRZU8CpSjyA) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663296290.3773835,"logger":"tls.obtain","msg":"will retry","error":"[sonarr.naff.casa] Obtain: [sonarr.naff.casa] solving challenges: [sonarr.naff.casa] authorization took too long (order=https://acme.zerossl.com/v2/DV90/order/PRSIphpSLbOmRZU8CpSjyA) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":332.521257746,"max_duration":2592000}
{"level":"info","ts":1663296290.3774164,"logger":"tls.obtain","msg":"releasing lock","identifier":"sonarr.naff.casa"}
{"level":"warn","ts":1663296290.3774254,"logger":"tls.issuance.acme.acme_client","msg":"HTTP request failed; retrying","url":"https://acme.zerossl.com/v2/DV90/authz/9_S1Y4yVI6Aro1BwJ9MG-w","error":"performing request: Post \"https://acme.zerossl.com/v2/DV90/authz/9_S1Y4yVI6Aro1BwJ9MG-w\": context canceled"}
{"level":"error","ts":1663296290.3774765,"logger":"tls.issuance.acme.acme_client","msg":"deactivating authorization","identifier":"remote.naff.casa","authz":"https://acme.zerossl.com/v2/DV90/authz/9_S1Y4yVI6Aro1BwJ9MG-w","error":"attempt 1: https://acme.zerossl.com/v2/DV90/authz/9_S1Y4yVI6Aro1BwJ9MG-w: context canceled"}
{"level":"error","ts":1663296290.3775291,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"remote.naff.casa","issuer":"acme.zerossl.com-v2-DV90","error":"[remote.naff.casa] solving challenges: [remote.naff.casa] context canceled (order=https://acme.zerossl.com/v2/DV90/order/2-Dj9X_G5jCTHKo10OthUQ) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663296290.37754,"logger":"tls.obtain","msg":"unable to unlock","identifier":"sonarr.naff.casa","lock_key":"issue_cert_sonarr.naff.casa","error":"remove /data/caddy/locks/issue_cert_sonarr.naff.casa.lock: no such file or directory"}
{"level":"info","ts":1663296290.3775606,"logger":"tls.obtain","msg":"releasing lock","identifier":"remote.naff.casa"}
{"level":"error","ts":1663296290.3775754,"logger":"tls","msg":"job failed","error":"sonarr.naff.casa: obtaining certificate: context canceled"}
{"level":"error","ts":1663296290.3776183,"logger":"tls.obtain","msg":"unable to unlock","identifier":"remote.naff.casa","lock_key":"issue_cert_remote.naff.casa","error":"remove /data/caddy/locks/issue_cert_remote.naff.casa.lock: no such file or directory"}
{"level":"error","ts":1663296290.3774486,"logger":"tls","msg":"job failed","error":"bw.naff.casa: obtaining certificate: context canceled"}
{"level":"error","ts":1663296290.3776488,"logger":"tls","msg":"job failed","error":"remote.naff.casa: obtaining certificate: [remote.naff.casa] Obtain: [remote.naff.casa] solving challenges: [remote.naff.casa] context canceled (order=https://acme.zerossl.com/v2/DV90/order/2-Dj9X_G5jCTHKo10OthUQ) (ca=https://acme.zerossl.com/v2/DV90)"}
{"level":"error","ts":1663296290.377668,"logger":"tls","msg":"job failed","error":"cctv.naff.casa: obtaining certificate: context canceled"}
{"level":"info","ts":1663296290.3785064,"logger":"admin","msg":"stopped previous server","address":"tcp/localhost:2019"}
{"level":"info","ts":1663296290.378583,"msg":"shutdown complete","signal":"SIGTERM","exit_code":0}
10

Hmm! Could be something ZeroSSL-specific, I note that it didn’t have a chance to try LetsEncrypt.

Spin it up again and watch the logs, but give it time to fail out from ZeroSSL and try LE as an issuer. Checking DNS from my end they go to the same place, so there shouldn’t be any problems with sonarr.naff.casa if e.g. plex.naff.casa worked fine.

11

Ok so ZeroSSL is different then LetsEncrypt. I thought I was using LetsEncrypt becuase that’s what I used with Traefik2 previously.

How long should I watch it? I don’t want to get banned from the server for too many requests.

12

all my other containers work except for these 4:

{"level":"info","ts":1663297447.0707777,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["sonarr.naff.casa"],"ca":"https://acme.zerossl.com/v2/DV90","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663297447.070837,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["sonarr.naff.casa"],"ca":"https://acme.zerossl.com/v2/DV90","account":"ryan.naff@gmail.com"}
{"level":"info","ts":1663297459.8784046,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"remote.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1663297463.7536745,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"bw.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1663297463.8107445,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"sonarr.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1663297463.9747725,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"cctv.naff.casa","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
13

Hmm, yeah, those are because it’s trying HTTP-01 challenges.

Per the previous thread, you can’t use those, because your ISP has blocked port 80 and this port must be externally accessible to receive HTTP-01 challenges.

So, we need TLS-ALPN.

I just had a quick look to see if there’s a global option to disable HTTP-01 challenges across the board (just to make Caddy stop trying them so it can get to the working TLS-ALPN-01 challenges quicker), but it looks like there’s no global for it. You’ll have to configure this with the tls directive on a per-site basis, I think.

Adding this block to your four problem sites should force Caddy to use LetsEncrypt and disable HTTP-01, give it a shot and see how you go:

tls {
  issuer acme https://acme-v02.api.letsencrypt.org/directory {
    disable_http_challenge
  }
}

(Per: tls (Caddyfile directive) — Caddy Documentation)

14

I don’t doubt what you’re saying isn’t true but I had all of these containers working fine previously, so seems weird that they aren’t now.

So I enter this code with EACH container, will it look like this?


}
tls {
  issuer acme https://acme-v02.api.letsencrypt.org/directory {
    disable_http_challenge
  }
}
sonarr.naff.casa {
    reverse_proxy 192.168.1.37:8989
15

They haven’t ‘stopped working’ per se, they’re just a little blocked in acquiring fresh certificates.

Normal Caddy operation involves attempting various ACME providers and challenge modes; ZeroSSL and LetsEncrypt are both options, and HTTP-01 and TLS-ALPN-01 challenge modes are available.

It’s most likely pure luck of the draw that the other containers were quickly cycled through to a working combination, but those four were not and have been attempting a method that is unavailable.

Normally this isn’t an issue since Caddy will eventually get around to trying other ways until it gets one that works, within reasonable rate limits, but this extra config will simply speed up that process by telling Caddy in no uncertain terms: “These are the parameters that will work, you can stop trying random methods.”

You can add these now, let the sites obtain their certificates, and then remove the configuration after, if you like; it’s purely to get Caddy over this first hump of first run obtaining certs.

The TLS block needs to be inside the relevant site, e.g.

example.com {
  # reverse_proxy config
  tls {
    issuer acme https://acme-v02.api.letsencrypt.org/directory {
      disable_http_challenge
    }
  }
}

And it must be individually present inside each site block that you need to configure this way (all four that are having problems).

16

I don’t understand the fundamentals of yamml and coding so I am wrecking this.

}
sonarr.naff.casa {
    reverse_proxy 192.168.1.37:8989
  tls
    issuer acme https://acme-v02.api.letsencrypt.org/directory {
      disable_http_challenge
    }
  }
}

}
plex.naff.casa {
17

The only fundamentals to worry about here are readability (e.g. indentation) and hierarchy (e.g. curly braces {}).

You’ve got, I think, one too many closing braces } and you’re missing one opening brace { after tls (that was my fault for not including it). It should look like:

sonarr.naff.casa {
  reverse_proxy 192.168.1.37:8989
  tls {
    issuer acme https://acme-v02.api.letsencrypt.org/directory {
      disable_http_challenge
    }
  }
}
18

it never ends:

{"level":"info","ts":1663301296.6384647,"logger":"tls","msg":"served key authentication certificate","server_name":"sonarr.naff.casa","challenge":"tls-alpn-01","remote":"23.178.112.107:36808","distributed":false}
{"level":"info","ts":1663301296.890077,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/68739064/4093262374"}
{"level":"info","ts":1663301297.1239588,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":1,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fad3d42c71e9f44ea3fe6c4e6ba05d775402"}
{"level":"error","ts":1663301297.2592115,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"sonarr.naff.casa","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/too-many-registrations-for-this-ip/"}
{"level":"error","ts":1663301297.2593157,"logger":"tls.obtain","msg":"will retry","error":"[sonarr.naff.casa] Obtain: registering account [mailto:ryan.naff@gmail.com] with server: attempt 1: https://acme-v02.api.letsencrypt.org/acme/new-acct: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new account :: too many registrations for this IP: see https://letsencrypt.org/docs/too-many-registrations-for-this-ip/","attempt":2,"retrying_in":120,"elapsed":64.823020561,"max_duration":2592000}
19

Huh!

The ‘registrations’ this error refers to are requests, sent from your IP address, to register a new account with the Let’s Encrypt API. This error indicates that at least 10 accounts have already been registered from this IP address in the last 3 hours.

—Registrations Per IP Limit - Let's Encrypt

Usually this might happen if you’re not preserving Caddy’s data properly between container starts, or if you’ve tried to set up more than just the one Caddy server recently (i.e. if you tried to migrate your Caddy server a lot more times than just once), and all this had happened quite recently.

That said, it’s a three hour rolling window for 10 account creations. Within a few hours you should be out from under that restriction and it should work fine again.

20

I am letting it run and it’s slowly figuring out the certs on it’s own.

Thanks again for all of your help!

1 Like