Error obtaining certificate after Caddy restart

1. Caddy version (caddy version):

v2.3.0 h1:fnrqJLa3G5vfxcxmOH/+kJOcunPLhSBnjgIvjXV/QTA=

2. How I run Caddy:

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/cfg/gpg/gpg.155B6D79CA56EA34.key' | sudo apt-key add -
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/cfg/setup/config.deb.txt?distro=debian&version=any-version' | sudo tee -a /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Ubuntu 20.10 x64
PHP 8.0.1 (cli) (built: Jan 13 2021 08:23:31) ( NTS )
Zend Engine v4.0.1, Copyright (c) Zend Technologies

b. Command:

sudo mkdir -p /var/www/html
sudo mkdir /var/log/caddy
sudo chown -R caddy:caddy /var/log/caddy
sudo nano /etc/caddy/Caddyfile

c. Service/unit/compose file:

NA

d. My complete Caddyfile or JSON config:

jatra.club {
    root * /var/www/html
    log {
        output file /var/log/caddy/access.log {
        	roll_size 3MiB
	        roll_keep 5
	        roll_keep_for 48h
        }
        format console
    }
    encode gzip zstd
    php_fastcgi unix//run/php/php8.0-fpm.sock
   
    tls my@email.com {
    	protocols tls1.2 tls1.3    	
    }
}

3. The problem I’m having:

My aim is to get my site to work on https://; however, it looks like Caddy is having trouble getting the certificate. I’ve pasted the full log output below.

Also, I’m unable to see any log entries in /var/log/caddy/access.log.

4. Error messages and/or full log output:

systemctl re
systemctl status caddy
â—Ź caddy.service - Caddy
     Loaded: loaded (/lib/systemd/system/caddy.service; enabled; vendor preset: enabled)
     Active: active (running) since Tue 2021-02-02 03:30:07 UTC; 11min ago
       Docs: https://caddyserver.com/docs/
   Main PID: 47867 (caddy)
      Tasks: 7 (limit: 1132)
     Memory: 14.5M
     CGroup: /system.slice/caddy.service
             └─47867 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

Feb 02 03:33:59 caddy caddy[47867]: {"level":"info","ts":1612236839.707516,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"jatra.club","challenge_type":"tls-alpn-01","ca":"https://acme-stag>
Feb 02 03:33:59 caddy caddy[47867]: {"level":"info","ts":1612236839.981753,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"18.224.20.83:14238","distribute>
Feb 02 03:33:59 caddy caddy[47867]: {"level":"info","ts":1612236839.9981575,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"66.133.109.36:61678","distribu>
Feb 02 03:34:00 caddy caddy[47867]: {"level":"info","ts":1612236840.0486758,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"34.211.60.134:59526","distribu>
Feb 02 03:34:10 caddy caddy[47867]: {"level":"info","ts":1612236850.1861618,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/>
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.2174249,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-staging-v02.api.>
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.220493,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.2210016,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.934589,"logger":"tls.obtain","msg":"releasing lock","identifier":"jatra.club"}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"error","ts":1612236851.9357755,"logger":"tls","msg":"job failed","error":"jatra.club: obtaining certificate: [jatra.club] Obtain: [jatra.club] creating new order: request to https://ac>

5. What I already tried:

I’ve been trying multiple configurations of Caddyfile with my domain jatra.club. Since I’ve had multiple failed attempts; I suspect letsencrypt has hit some rate limiting on my domain.

Can someone advise me how to fix this? Thanks!

6. Links to relevant resources:

This is the tutorial I followed:

Please use journalctl -u caddy --no-pager | less to view your logs, because they’re otherwise truncated (see the > at the end of most lines). The actual error message is truncated so we can’t see what it says.

I wish people would stop copying or rewriting the same content that’s on the official docs, and would instead link there. It’s only asking to be made out of date. Case-in-point, it still recommends using gemfury, which is no longer our recommendation. It also has outdated Caddyfile config and pointless bits.

You can remove the protocols stuff and the braces. It has no effect, because those are already the defaults.

format json is more useful, because you’ll be able to parse it with jq. The console format doesn’t really make sense for file output.

Also, you seem to be missing the file_server directive, which is necessary for serving any static assets your site may have (like .js or .css files)

2 Likes

@francislavoie - here’s the output of journalctl -u caddy --no-pager

-- Logs begin at Tue 2021-02-02 03:11:18 UTC, end at Tue 2021-02-02 14:14:52 UTC. --
Feb 02 03:20:06 caddy systemd[1]: Started Caddy.
Feb 02 03:20:06 caddy caddy[32130]: caddy.HomeDir=/var/lib/caddy
Feb 02 03:20:06 caddy caddy[32130]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 02 03:20:06 caddy caddy[32130]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 02 03:20:06 caddy caddy[32130]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 02 03:20:06 caddy caddy[32130]: caddy.Version=v2.3.0
Feb 02 03:20:06 caddy caddy[32130]: runtime.GOOS=linux
Feb 02 03:20:06 caddy caddy[32130]: runtime.GOARCH=amd64
Feb 02 03:20:06 caddy caddy[32130]: runtime.Compiler=gc
Feb 02 03:20:06 caddy caddy[32130]: runtime.NumCPU=1
Feb 02 03:20:06 caddy caddy[32130]: runtime.GOMAXPROCS=1
Feb 02 03:20:06 caddy caddy[32130]: runtime.Version=go1.15.6
Feb 02 03:20:06 caddy caddy[32130]: os.Getwd=/
Feb 02 03:20:06 caddy caddy[32130]: LANG=C.UTF-8
Feb 02 03:20:06 caddy caddy[32130]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Feb 02 03:20:06 caddy caddy[32130]: HOME=/var/lib/caddy
Feb 02 03:20:06 caddy caddy[32130]: LOGNAME=caddy
Feb 02 03:20:06 caddy caddy[32130]: USER=caddy
Feb 02 03:20:06 caddy caddy[32130]: INVOCATION_ID=241dd21b0347418c82b293b2e6879e44
Feb 02 03:20:06 caddy caddy[32130]: JOURNAL_STREAM=8:68772
Feb 02 03:20:06 caddy caddy[32130]: {"level":"info","ts":1612236006.3053615,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 02 03:20:06 caddy caddy[32130]: {"level":"info","ts":1612236006.312994,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Feb 02 03:20:06 caddy caddy[32130]: {"level":"info","ts":1612236006.313655,"logger":"http","msg":"server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server","server_name":"srv0","http_port":80}
Feb 02 03:20:06 caddy caddy[32130]: {"level":"info","ts":1612236006.3142915,"logger":"tls","msg":"cleaned up storage units"}
Feb 02 03:20:06 caddy caddy[32130]: {"level":"info","ts":1612236006.314637,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000227c70"}
Feb 02 03:20:06 caddy caddy[32130]: {"level":"info","ts":1612236006.3172433,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 02 03:20:06 caddy caddy[32130]: {"level":"info","ts":1612236006.3175137,"msg":"serving initial configuration"}
Feb 02 03:30:06 caddy caddy[32130]: {"level":"info","ts":1612236606.5752962,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
Feb 02 03:30:06 caddy systemd[1]: Stopping Caddy...
Feb 02 03:30:07 caddy caddy[32130]: {"level":"info","ts":1612236607.0761116,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc000227c70"}
Feb 02 03:30:07 caddy caddy[32130]: {"level":"info","ts":1612236607.5766149,"logger":"admin","msg":"stopped previous server"}
Feb 02 03:30:07 caddy caddy[32130]: {"level":"info","ts":1612236607.5767636,"msg":"shutdown done","signal":"SIGTERM"}
Feb 02 03:30:07 caddy systemd[1]: caddy.service: Succeeded.
Feb 02 03:30:07 caddy systemd[1]: Stopped Caddy.
Feb 02 03:30:07 caddy systemd[1]: Started Caddy.
Feb 02 03:30:07 caddy caddy[47867]: caddy.HomeDir=/var/lib/caddy
Feb 02 03:30:07 caddy caddy[47867]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 02 03:30:07 caddy caddy[47867]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 02 03:30:07 caddy caddy[47867]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 02 03:30:07 caddy caddy[47867]: caddy.Version=v2.3.0
Feb 02 03:30:07 caddy caddy[47867]: runtime.GOOS=linux
Feb 02 03:30:07 caddy caddy[47867]: runtime.GOARCH=amd64
Feb 02 03:30:07 caddy caddy[47867]: runtime.Compiler=gc
Feb 02 03:30:07 caddy caddy[47867]: runtime.NumCPU=1
Feb 02 03:30:07 caddy caddy[47867]: runtime.GOMAXPROCS=1
Feb 02 03:30:07 caddy caddy[47867]: runtime.Version=go1.15.6
Feb 02 03:30:07 caddy caddy[47867]: os.Getwd=/
Feb 02 03:30:07 caddy caddy[47867]: LANG=C.UTF-8
Feb 02 03:30:07 caddy caddy[47867]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Feb 02 03:30:07 caddy caddy[47867]: HOME=/var/lib/caddy
Feb 02 03:30:07 caddy caddy[47867]: LOGNAME=caddy
Feb 02 03:30:07 caddy caddy[47867]: USER=caddy
Feb 02 03:30:07 caddy caddy[47867]: INVOCATION_ID=ce14dbec1f1f4fe7919804a577713472
Feb 02 03:30:07 caddy caddy[47867]: JOURNAL_STREAM=8:92496
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.7156017,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.723631,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.7249339,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.7267861,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jatra.club"]}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.7277899,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.728167,"msg":"serving initial configuration"}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.728645,"logger":"tls","msg":"cleaned up storage units"}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.7292526,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0002132d0"}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.7307208,"logger":"tls.obtain","msg":"acquiring lock","identifier":"jatra.club"}
Feb 02 03:30:07 caddy caddy[47867]: {"level":"info","ts":1612236607.7316804,"logger":"tls.obtain","msg":"lock acquired","identifier":"jatra.club"}
Feb 02 03:30:08 caddy caddy[47867]: {"level":"info","ts":1612236608.7494824,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:30:08 caddy caddy[47867]: {"level":"info","ts":1612236608.7508113,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:30:08 caddy caddy[47867]: {"level":"error","ts":1612236608.9008844,"logger":"tls.obtain","msg":"will retry","error":"[jatra.club] Obtain: [jatra.club] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates already issued for exact set of domains: jatra.club: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":1.16886995,"max_duration":2592000}
Feb 02 03:31:10 caddy caddy[47867]: {"level":"info","ts":1612236670.0833192,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"jatra.club","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Feb 02 03:31:13 caddy caddy[47867]: {"level":"info","ts":1612236673.5773263,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"66.133.109.36:22898","distributed":false}
Feb 02 03:31:20 caddy caddy[47867]: {"level":"info","ts":1612236680.5153778,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"34.211.60.134:57964","distributed":false}
Feb 02 03:31:40 caddy caddy[47867]: {"level":"error","ts":1612236700.606927,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"jatra.club","challenge_type":"tls-alpn-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:dns","error":"During secondary validation: No valid IP addresses found for jatra.club"}
Feb 02 03:31:40 caddy caddy[47867]: {"level":"error","ts":1612236700.6090877,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"jatra.club","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - During secondary validation: No valid IP addresses found for jatra.club","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17873356/232459776","attempt":1,"max_attempts":3}
Feb 02 03:31:41 caddy caddy[47867]: {"level":"info","ts":1612236701.916529,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"jatra.club","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Feb 02 03:31:51 caddy caddy[47867]: {"level":"error","ts":1612236711.929771,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"jatra.club","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"Fetching https://jatra.club/.well-known/acme-challenge/HJw8cDgSITOej7KerPNirdrqQUZOfpP5kWKyVth5GWU: remote error: tls: internal error"}
Feb 02 03:31:51 caddy caddy[47867]: {"level":"error","ts":1612236711.931161,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"jatra.club","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - Fetching https://jatra.club/.well-known/acme-challenge/HJw8cDgSITOej7KerPNirdrqQUZOfpP5kWKyVth5GWU: remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17873356/232459977","attempt":2,"max_attempts":3}
Feb 02 03:31:53 caddy caddy[47867]: {"level":"error","ts":1612236713.4272418,"logger":"tls.obtain","msg":"will retry","error":"[jatra.club] Obtain: [jatra.club] solving challenges: jatra.club: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-staging-v02.api.letsencrypt.org/acme/order/17873356/232460066) (ca=https://acme-staging-v02.api.letsencrypt.org/directory)","attempt":2,"retrying_in":120,"elapsed":105.695227714,"max_duration":2592000}
Feb 02 03:31:54 caddy caddy[47867]: {"level":"info","ts":1612236714.6308272,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:43292","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1412"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Feb 02 03:31:54 caddy caddy[47867]: {"level":"info","ts":1612236714.6314454,"logger":"admin.api","msg":"config is unchanged"}
Feb 02 03:31:54 caddy caddy[47867]: {"level":"info","ts":1612236714.631466,"logger":"admin.api","msg":"load complete"}
Feb 02 03:33:54 caddy caddy[47867]: {"level":"info","ts":1612236834.30535,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"jatra.club","challenge_type":"http-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Feb 02 03:33:58 caddy caddy[47867]: {"level":"error","ts":1612236838.3977983,"logger":"tls.issuance.acme.acme_client","msg":"challenge failed","identifier":"jatra.club","challenge_type":"http-01","status_code":400,"problem_type":"urn:ietf:params:acme:error:tls","error":"Fetching https://jatra.club/.well-known/acme-challenge/l-mF_WgiqDH0DRzxT2tHW0MD8a9xcjK6m1V32qV-FA4: remote error: tls: internal error"}
Feb 02 03:33:58 caddy caddy[47867]: {"level":"error","ts":1612236838.3990538,"logger":"tls.issuance.acme.acme_client","msg":"validating authorization","identifier":"jatra.club","error":"authorization failed: HTTP 400 urn:ietf:params:acme:error:tls - Fetching https://jatra.club/.well-known/acme-challenge/l-mF_WgiqDH0DRzxT2tHW0MD8a9xcjK6m1V32qV-FA4: remote error: tls: internal error","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17873356/232460942","attempt":1,"max_attempts":3}
Feb 02 03:33:59 caddy caddy[47867]: {"level":"info","ts":1612236839.707516,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"jatra.club","challenge_type":"tls-alpn-01","ca":"https://acme-staging-v02.api.letsencrypt.org/directory"}
Feb 02 03:33:59 caddy caddy[47867]: {"level":"info","ts":1612236839.981753,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"18.224.20.83:14238","distributed":false}
Feb 02 03:33:59 caddy caddy[47867]: {"level":"info","ts":1612236839.9981575,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"66.133.109.36:61678","distributed":false}
Feb 02 03:34:00 caddy caddy[47867]: {"level":"info","ts":1612236840.0486758,"logger":"tls","msg":"served key authentication certificate","server_name":"jatra.club","challenge":"tls-alpn-01","remote":"34.211.60.134:59526","distributed":false}
Feb 02 03:34:10 caddy caddy[47867]: {"level":"info","ts":1612236850.1861618,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17873356/232460991"}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.2174249,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fa2a13d88aff1432746df7c75f9fc6698059"}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.220493,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.2210016,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"info","ts":1612236851.934589,"logger":"tls.obtain","msg":"releasing lock","identifier":"jatra.club"}
Feb 02 03:34:11 caddy caddy[47867]: {"level":"error","ts":1612236851.9357755,"logger":"tls","msg":"job failed","error":"jatra.club: obtaining certificate: [jatra.club] Obtain: [jatra.club] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates already issued for exact set of domains: jatra.club: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Feb 02 03:46:04 caddy caddy[47867]: {"level":"info","ts":1612237564.253158,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
Feb 02 03:46:04 caddy systemd[1]: Stopping Caddy...
Feb 02 03:46:05 caddy caddy[47867]: {"level":"info","ts":1612237565.25441,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc0002132d0"}
Feb 02 03:46:05 caddy caddy[47867]: {"level":"info","ts":1612237565.7547705,"logger":"admin","msg":"stopped previous server"}
Feb 02 03:46:05 caddy caddy[47867]: {"level":"info","ts":1612237565.7548537,"msg":"shutdown done","signal":"SIGTERM"}
Feb 02 03:46:05 caddy systemd[1]: caddy.service: Succeeded.
Feb 02 03:46:05 caddy systemd[1]: Stopped Caddy.
Feb 02 03:46:05 caddy systemd[1]: Started Caddy.
Feb 02 03:46:05 caddy caddy[48246]: caddy.HomeDir=/var/lib/caddy
Feb 02 03:46:05 caddy caddy[48246]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 02 03:46:05 caddy caddy[48246]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 02 03:46:05 caddy caddy[48246]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 02 03:46:05 caddy caddy[48246]: caddy.Version=v2.3.0
Feb 02 03:46:05 caddy caddy[48246]: runtime.GOOS=linux
Feb 02 03:46:05 caddy caddy[48246]: runtime.GOARCH=amd64
Feb 02 03:46:05 caddy caddy[48246]: runtime.Compiler=gc
Feb 02 03:46:05 caddy caddy[48246]: runtime.NumCPU=1
Feb 02 03:46:05 caddy caddy[48246]: runtime.GOMAXPROCS=1
Feb 02 03:46:05 caddy caddy[48246]: runtime.Version=go1.15.6
Feb 02 03:46:05 caddy caddy[48246]: os.Getwd=/
Feb 02 03:46:05 caddy caddy[48246]: LANG=C.UTF-8
Feb 02 03:46:05 caddy caddy[48246]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Feb 02 03:46:05 caddy caddy[48246]: HOME=/var/lib/caddy
Feb 02 03:46:05 caddy caddy[48246]: LOGNAME=caddy
Feb 02 03:46:05 caddy caddy[48246]: USER=caddy
Feb 02 03:46:05 caddy caddy[48246]: INVOCATION_ID=cd3c69947dc34952a23727303f5b60e8
Feb 02 03:46:05 caddy caddy[48246]: JOURNAL_STREAM=8:98286
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8655524,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.871021,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8723476,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8732917,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00022ba40"}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8748314,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jatra.club"]}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8754754,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8757238,"msg":"serving initial configuration"}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8765278,"logger":"tls.obtain","msg":"acquiring lock","identifier":"jatra.club"}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.8772423,"logger":"tls.obtain","msg":"lock acquired","identifier":"jatra.club"}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.9009528,"logger":"tls","msg":"cleaned up storage units"}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.9164593,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:46:05 caddy caddy[48246]: {"level":"info","ts":1612237565.917114,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:46:06 caddy caddy[48246]: {"level":"error","ts":1612237566.8677936,"logger":"tls.obtain","msg":"will retry","error":"[jatra.club] Obtain: [jatra.club] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates already issued for exact set of domains: jatra.club: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.99027743,"max_duration":2592000}
Feb 02 03:47:07 caddy caddy[48246]: {"level":"info","ts":1612237627.892348,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme-staging-v02.api.letsencrypt.org/acme/order/17873356/232466234"}
Feb 02 03:47:09 caddy caddy[48246]: {"level":"info","ts":1612237629.0023158,"logger":"tls.issuance.acme.acme_client","msg":"successfully downloaded available certificate chains","count":2,"first_url":"https://acme-staging-v02.api.letsencrypt.org/acme/cert/fafb1cc7687b0e34288923b7883172fbd4fb"}
Feb 02 03:47:09 caddy caddy[48246]: {"level":"info","ts":1612237629.004835,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:47:09 caddy caddy[48246]: {"level":"info","ts":1612237629.0053704,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 03:47:09 caddy caddy[48246]: {"level":"info","ts":1612237629.7195256,"logger":"tls.obtain","msg":"releasing lock","identifier":"jatra.club"}
Feb 02 03:47:09 caddy caddy[48246]: {"level":"error","ts":1612237629.7210603,"logger":"tls","msg":"job failed","error":"jatra.club: obtaining certificate: [jatra.club] Obtain: [jatra.club] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates already issued for exact set of domains: jatra.club: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Feb 02 14:14:05 caddy caddy[48246]: {"level":"info","ts":1612275245.8601503,"msg":"shutting down apps then terminating","signal":"SIGTERM"}
Feb 02 14:14:05 caddy systemd[1]: Stopping Caddy...
Feb 02 14:14:06 caddy caddy[48246]: {"level":"info","ts":1612275246.860959,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00022ba40"}
Feb 02 14:14:07 caddy caddy[48246]: {"level":"info","ts":1612275247.3613584,"logger":"admin","msg":"stopped previous server"}
Feb 02 14:14:07 caddy caddy[48246]: {"level":"info","ts":1612275247.3614342,"msg":"shutdown done","signal":"SIGTERM"}
Feb 02 14:14:07 caddy systemd[1]: caddy.service: Succeeded.
Feb 02 14:14:07 caddy systemd[1]: Stopped Caddy.
-- Reboot --
Feb 02 14:14:39 caddy systemd[1]: Started Caddy.
Feb 02 14:14:41 caddy caddy[596]: caddy.HomeDir=/var/lib/caddy
Feb 02 14:14:41 caddy caddy[596]: caddy.AppDataDir=/var/lib/caddy/.local/share/caddy
Feb 02 14:14:41 caddy caddy[596]: caddy.AppConfigDir=/var/lib/caddy/.config/caddy
Feb 02 14:14:41 caddy caddy[596]: caddy.ConfigAutosavePath=/var/lib/caddy/.config/caddy/autosave.json
Feb 02 14:14:41 caddy caddy[596]: caddy.Version=v2.3.0
Feb 02 14:14:41 caddy caddy[596]: runtime.GOOS=linux
Feb 02 14:14:41 caddy caddy[596]: runtime.GOARCH=amd64
Feb 02 14:14:41 caddy caddy[596]: runtime.Compiler=gc
Feb 02 14:14:41 caddy caddy[596]: runtime.NumCPU=1
Feb 02 14:14:41 caddy caddy[596]: runtime.GOMAXPROCS=1
Feb 02 14:14:41 caddy caddy[596]: runtime.Version=go1.15.6
Feb 02 14:14:41 caddy caddy[596]: os.Getwd=/
Feb 02 14:14:41 caddy caddy[596]: LANG=C.UTF-8
Feb 02 14:14:41 caddy caddy[596]: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/snap/bin
Feb 02 14:14:41 caddy caddy[596]: HOME=/var/lib/caddy
Feb 02 14:14:41 caddy caddy[596]: LOGNAME=caddy
Feb 02 14:14:41 caddy caddy[596]: USER=caddy
Feb 02 14:14:41 caddy caddy[596]: INVOCATION_ID=d753e7bd617941b2b91d69eb23bcfa2a
Feb 02 14:14:41 caddy caddy[596]: JOURNAL_STREAM=8:22855
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.5257733,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.552484,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.5718653,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.5734947,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jatra.club"]}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.587418,"logger":"tls","msg":"cleaned up storage units"}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.592777,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00024f7a0"}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.5985184,"logger":"tls.obtain","msg":"acquiring lock","identifier":"jatra.club"}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.6144493,"logger":"tls.obtain","msg":"lock acquired","identifier":"jatra.club"}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.6303127,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.630724,"msg":"serving initial configuration"}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.6964347,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 14:14:41 caddy caddy[596]: {"level":"info","ts":1612275281.6968062,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 02 14:14:43 caddy caddy[596]: {"level":"error","ts":1612275283.7961617,"logger":"tls.obtain","msg":"will retry","error":"[jatra.club] Obtain: [jatra.club] creating new order: request to https://acme-v02.api.letsencrypt.org/acme/new-order failed after 1 attempts: HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates already issued for exact set of domains: jatra.club: see https://letsencrypt.org/docs/rate-limits/ (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":2.181409464,"max_duration":2592000}

I think I’ve found the error - “Error creating new order :: too many certificates already issued for exact set of domains: jatra.club”

I’d like to know what’s the right way to avoid such error in future? Also, I think the following Caddyfile configuration looks right. I can’t test it because of above error though -

jatra.club {
    root * /var/www/html
    log {
        output file /var/log/caddy/access.log {
        	roll_size 3MiB
	        roll_keep 5
	        roll_keep_for 48h
        }
        format json #added
    }
    encode gzip zstd
    php_fastcgi unix//run/php/php8.0-fpm.sock
    file_server #added
}

Would it work if I replaced jatra.club with jatra.club, *.jatra.club? My goal is to let users have their own subdomains.

Thank you for the incredible support so far!

I don’t understand how you might’ve hit that rate limit. If you’re running directly on Ubuntu as a systemd service, and not in a Docker container, then Caddy’s certificates should be persisted no problem. You should find them in /var/lib/caddy/.local/share/caddy

Have you been throwing away your machine/VM/VPS and restarting frequently in the past while? Do you have many servers using the Caddy with the same domain, or did you use certbot or something else before?

The rate limits are listed here, and I think you hit the 5 duplicate certificates per week limit:

When Caddy fails to fetch a certificate with let’s encrypt, it’ll then try with the staging endpoint as a fallback just to see if things are correctly configured for your server before trying again. Looks like some of the staging attempts are failing because let’s encrypt can’t reach your server. For example:

What I find strange though is that Caddy didn’t seem to attempt to try with ZeroSSL which is an alternate ACME provider which you would not have been rate limited against probably. @matt do the logs give you any hint as to why?

Yes, that’s what happened. I’m on DigitalOcean and I attempted installation and configuration on several droplets one after the other. I’ve an array of droplets that I use for testing our application.

I think that could have led to LetsEncrypt hitting the limits.

I believe that error could have been from the event that LetsEncrypt was unable to reach my droplet’s A address. But that’s my wild guess.

Is there any way I could know when will I have access to the certificate for my domain?
Update: Found it here-: https://crt.sh/ .

Yeah, best not to do that. If you’re repeatedly throwing away droplets, make sure to explicitly tell Caddy to use the LE staging endpoint while you’re testing to make sure you don’t hit rate limits.

You’d have to find the first time you were served with that error; then it would be one week from that date. If you don’t still have the logs, then it should be somewhere less than a week, I figure.

Something you could try as a workaround, is explicitly configuring zerossl to force Caddy to use it. I’m not sure why it didn’t use it as a fallback for you, since you’re on v2.3.0 and that’s how it should behave from what I understand of the changes that were introduced.

I think adding this to your site block should force it to use ZeroSSL:

tls {
	issuer zerossl
}

Once you have a certificate, you should be able to remove that bit from your config and reload Caddy, because once you have a valid certificate, Caddy will reuse the same issuer for renewals. It’s best to leave both as possible issuers, in case either one goes down for whatever reason (reduces the points of failure).

1 Like

Realized I didn’t answer this question – if you plan to have wildcard certs, you’ll need to configure the ACME DNS challenge. See here:

1 Like

I suspect that tls <email> overrides the default issuers, replacing it with just Let’s Encrypt. I haven’t thought about whether that is what should be expected or not.

1 Like

I fixed the multi-CA fallback in the case of providing an email address in the tls subdirective, here: httpcaddyfile: Fix default issuers when email provided · caddyserver/caddy@90284e8 · GitHub

@francislavoie : I changed the Caddyfile to use zerossl and it seems to work as expected:

New Caddyfile Configuration:

jatra.club {
    root * /var/www/html
    log {
        output file /var/log/caddy/access.log {
		roll_size 3MiB
	        roll_keep 5
	        roll_keep_for 48h
        }
        format json
    }
	encode gzip zstd
	php_fastcgi unix//run/php/php8.0-fpm.sock
	tls {
		issuer zerossl
	}	
	file_server
}

Logs:

Feb 03 03:09:54 caddy systemd[1]: Reloading Caddy.
Feb 03 03:09:54 caddy caddy[4690]: {"level":"info","ts":1612321794.5879896,"msg":"using provided configuration","config_file":"/etc/caddy/Caddyfile","config_adapter":""}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.5921032,"logger":"admin.api","msg":"received request","method":"POST","host":"localhost:2019","uri":"/load","remote_addr":"127.0.0.1:35162","headers":{"Accept-Encoding":["gzip"],"Content-Length":["1340"],"Content-Type":["application/json"],"Origin":["localhost:2019"],"User-Agent":["Go-http-client/1.1"]}}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.5928128,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.5932298,"logger":"http","msg":"server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS","server_name":"srv0","https_port":443}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.593252,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"srv0"}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.593894,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["jatra.club"]}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.5950246,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000e6230"}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.5954497,"logger":"tls","msg":"cleaned up storage units"}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.5961242,"logger":"tls.obtain","msg":"acquiring lock","identifier":"jatra.club"}
Feb 03 03:09:54 caddy caddy[596]: {"level":"info","ts":1612321794.596518,"logger":"tls.obtain","msg":"lock acquired","identifier":"jatra.club"}
Feb 03 03:09:54 caddy caddy[596]: {"level":"warn","ts":1612321794.5974045,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
Feb 03 03:09:55 caddy caddy[596]: {"level":"info","ts":1612321795.595178,"logger":"tls.cache.maintenance","msg":"stopped background certificate maintenance","cache":"0xc00024f7a0"}
Feb 03 03:09:55 caddy caddy[596]: {"level":"info","ts":1612321795.5965192,"msg":"autosaved config","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Feb 03 03:09:55 caddy caddy[596]: {"level":"info","ts":1612321795.5967884,"logger":"admin.api","msg":"load complete"}
Feb 03 03:09:55 caddy systemd[1]: Reloaded Caddy.
Feb 03 03:09:55 caddy caddy[596]: {"level":"info","ts":1612321795.6046672,"logger":"admin","msg":"stopped previous server"}
Feb 03 03:09:55 caddy caddy[596]: {"level":"info","ts":1612321795.8745897,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"rWXC2Z-x6r44uPVsfsGyoQ"}
Feb 03 03:09:56 caddy caddy[596]: {"level":"info","ts":1612321796.1766768,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 03 03:09:56 caddy caddy[596]: {"level":"info","ts":1612321796.1772685,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["jatra.club"]}
Feb 03 03:09:56 caddy caddy[596]: {"level":"info","ts":1612321796.3485346,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"jatra.club","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
Feb 03 03:09:56 caddy caddy[596]: {"level":"info","ts":1612321796.697901,"logger":"tls.issuance.acme","msg":"served key authentication","identifier":"jatra.club","challenge":"http-01","remote":"91.199.212.132:36004"}
Feb 03 03:09:56 caddy caddy[596]: {"level":"info","ts":1612321796.7430005,"logger":"tls.issuance.acme.acme_client","msg":"validations succeeded; finalizing order","order":"https://acme.zerossl.com/v2/DV90/order/KsKyij1LWLjbt6eCSGo9_Q"}

Last line shows that Caddy hit the ZeroSSL endpoint as expected.

My next challenge is figuring out wildcard subdomains and mapped domains served using SSL.

I really appreciate your quick help @francislavoie and @matt :+1: :+1: :+1:

PS: Log files are working as well! :smiley:

1 Like

Do you mean domains that your customers control, that they point to your servers? In that case you’re looking for Caddy’s On-Demand TLS feature.

1 Like

Yes, that’s right. On-Demand TLS is what brought me to Caddy from NGINX. I want to let my users point their own domains or subdomains to their <subdomain>.jatra.club. I believe that’s exactly what the On-Demand TLS lets you do.

I’m hell scared to hit the certificate limits, though. I’ve never used ZeroSSL. Is there any gotcha’s I need to be aware of while trying this out?

Don’t forget to set an email address on your config so ZeroSSL can reach you if they have any problems or so you can use their dashboard features.

Also, you won’t hit rate limits with Let’s Encrypt if you use a wildcard for the subdomains of your domain. And don’t delete the certs from disk of course.

1 Like

Yeah. I recommend using a wildcard cert for *.jatra.club, and On-Demand TLS for the rest.

Nope. ZeroSSL actually has more loose rate limits than Let’s Encrypt. You should be able to safely remove the tls config from your Caddyfile now that you have a valid certificate, and let Caddy pick the appropriate one from here on.

Since your cert for jatra.club expires in 90 days (will be renewed in 2/3 of its life, in 60 days), you’re safe on rate limits now, since Caddy has no reason to attempt to fetch another certificate from LE for that domain (for the next 60 days).

Basically I’d recommend a config like this:

{
	on_demand_tls {
		ask https://example.com/ask
	}
}

jatra.club, *.jatra.club {
	tls {
		dns <provider>
	}

	...
}

https:// {
	tls {
		on_demand
	}

	...
}

You’ll have to fill in the blanks with the rest.

Basically I recommend you configure the DNS challenge for your main site. I don’t know which DNS provider you have, so you’ll need to build Caddy with the appropriate DNS plugin.

The https:// site is what will handle your On-Demand TLS. Any requests to domains that aren’t jatra.club will hit that site block, and have a certificate issued on the fly, given that the ask endpoint says that domain is good.

You’ll need to set up an ask endpoint, i.e. an API endpoint at which Caddy can ask your app to check its database for a given domain – this is to prevent abuse. You’ll need to have your users tell you the domain they plan to use beforehand, and you’ll add it to your database.

Anyways most of this is covered in this page:

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.