Need help to config ZeroSSL in caddy.yaml

1. The problem I’m having:

I need to config Caddy to work with my Livekit Server. I set up follow Livekit Docs but I stuck on configuring caddy.yaml. I have had own SSL Certs, but I found post below (I put in relevant resources) so I try to use ZeroSSL, it seems not work with error logs below.

In my server I have already used nginx for other processes. So does it make any conflicts?

Thank you!

2. Error messages and/or full log output:

{"level":"info","ts":1717155753.157638,"msg":"using provided configuration","config_file":"/etc/caddy.yaml","config_adapter":"yaml"}
{"level":"info","ts":1717155753.1610882,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1717155753.1618733,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00049a850"}
{"level":"info","ts":1717155753.1632066,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
{"level":"info","ts":1717155753.1632266,"msg":"serving initial configuration"}
{"level":"info","ts":1717155753.1632597,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data"}
{"level":"info","ts":1717155753.1634846,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"info","ts":1717155753.1640186,"logger":"tls.obtain","msg":"acquiring lock","identifier":"video.pancake.vn"}
{"level":"info","ts":1717155753.1647782,"logger":"tls.obtain","msg":"acquiring lock","identifier":"turnchat.pancake.vn"}
{"level":"info","ts":1717155753.1721122,"logger":"tls.obtain","msg":"lock acquired","identifier":"turnchat.pancake.vn"}
{"level":"info","ts":1717155753.172173,"logger":"tls.obtain","msg":"lock acquired","identifier":"video.pancake.vn"}
{"level":"info","ts":1717155753.172842,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"turnchat.pancake.vn"}
{"level":"info","ts":1717155753.1730306,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"video.pancake.vn"}
{"level":"info","ts":1717155753.1761513,"logger":"tls.issuance.zerossl","msg":"waiting on internal rate limiter","identifiers":["video.pancake.vn"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1717155753.1763527,"logger":"tls.issuance.zerossl","msg":"done waiting on internal rate limiter","identifiers":["video.pancake.vn"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1717155753.179512,"logger":"tls.issuance.zerossl","msg":"waiting on internal rate limiter","identifiers":["turnchat.pancake.vn"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1717155753.179562,"logger":"tls.issuance.zerossl","msg":"done waiting on internal rate limiter","identifiers":["turnchat.pancake.vn"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
{"level":"info","ts":1717155755.696537,"logger":"tls.issuance.zerossl.acme_client","msg":"trying to solve challenge","identifier":"video.pancake.vn","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}
{"level":"info","ts":1717155755.713596,"logger":"tls.issuance.zerossl.acme_client","msg":"trying to solve challenge","identifier":"turnchat.pancake.vn","challenge_type":"http-01","ca":"https://acme.zerossl.com/v2/DV90"}

3. Caddy version:

Caddy v2.6.2

4. How I installed and ran Caddy:

I run Caddy through Docker Image: https://hub.docker.com/r/livekit/caddyl4/tags

a. System environment:

Ubuntu 20.04.3

b. Command:

command: run --config /etc/caddy.yaml --adapter yaml

c. Service/unit/compose file:

version: "3.9"
services:
  caddy:
    image: livekit/caddyl4
    command: run --config /etc/caddy.yaml --adapter yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./caddy.yaml:/etc/caddy.yaml
      - ./caddy_data:/data
  livekit:
    image: livekit/livekit-server:latest
    command: --config /etc/livekit.yaml
    restart: unless-stopped
    network_mode: "host"
    volumes:
      - ./livekit.yaml:/etc/livekit.yaml
  redis:
    image: redis:6-alpine
    command: redis-server /etc/redis.conf
    network_mode: "host"
    volumes:
      - ./redis.conf:/etc/redis.conf

d. My complete Caddy config:

logging:
  logs:
    default:
      level: INFO
storage:
  "module": "file_system"
  "root": "/data"
apps:
  tls:
    certificates:
      automate:
        - video.pancake.vn
        - turnchat.pancake.vn
    automation:
      policies:
        - issuers:
          - module: zerossl
            api_key: my_api_key_from_zerossl
  layer4:
    servers:
      main:
        listen: [":4433"]
        routes:
          - match:
            - tls:
                sni:
                  - "turnchat.pancake.vn"
            handle:
              - handler: tls
              - handler: proxy
                upstreams:
                  - dial: ["10.1.13.40:5349"]
          - match:
              - tls:
                  sni:
                    - "video.pancake.vn"
            handle:
              - handler: tls
                connection_policies:
                  - alpn: ["http/1.1"]
              - handler: proxy
                upstreams:
                  - dial: ["localhost:7880"]

5. Links to relevant resources:

That’s quite an old version. Please use the latest version.

Thank for your response!
I try to use version Caddy Image ver 2.8.1. It seems can get certs from zerossl, it’s my error logs:

{"level":"info","ts":1717412941.6063242,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"turnchat.pancake.vn"}
{"level":"info","ts":1717412941.6072705,"logger":"tls.issuance.zerossl","msg":"creating certificate","identifiers":["turnchat.pancake.vn"]}
{"level":"info","ts":1717412942.9589767,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"video.pancake.vn"}
{"level":"info","ts":1717412942.9598875,"logger":"tls.issuance.zerossl","msg":"creating certificate","identifiers":["video.pancake.vn"]}
{"level":"info","ts":1717412944.5642345,"logger":"tls.issuance.zerossl","msg":"created certificate","identifiers":["turnchat.pancake.vn"],"cert_id":"bd60abc3077f80f689d93b32d315023d"}
{"level":"info","ts":1717412944.5652514,"logger":"tls.issuance.zerossl","msg":"validating identifiers","identifiers":["turnchat.pancake.vn"],"cert_id":"bd60abc3077f80f689d93b32d315023d","verification_method":"HTTP_CSR_HASH"}
{"level":"info","ts":1717412944.8363333,"logger":"tls.issuance.zerossl","msg":"created certificate","identifiers":["video.pancake.vn"],"cert_id":"17423b951963093e86c274f8483f9af5"}
{"level":"info","ts":1717412944.8373754,"logger":"tls.issuance.zerossl","msg":"validating identifiers","identifiers":["video.pancake.vn"],"cert_id":"17423b951963093e86c274f8483f9af5","verification_method":"HTTP_CSR_HASH"}
{"level":"info","ts":1717412945.828761,"logger":"tls.issuance.zerossl","msg":"canceled certificate","identifiers":["turnchat.pancake.vn"],"cert_id":"bd60abc3077f80f689d93b32d315023d","verification_method":"HTTP_CSR_HASH"}
{"level":"error","ts":1717412945.8288171,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"turnchat.pancake.vn","issuer":"zerossl","error":"verifying identifiers: POST https://api.zerossl.com/certificates/bd60abc3077f80f689d93b32d315023d/challenges?access_key=redacted: HTTP 200: API error 0: domain_control_validation_failed (details=map[turnchat.pancake.vn:map[http://turnchat.pancake.vn/.well-known/pki-validation/EEB6CD0CAA256FB76E4BCB825059E916.txt:{{0 0   } {0 true bad_response_code Server responded with status code: 404 (200 expected)}}]]) (raw={\"success\":false,\"error\":{\"code\":0,\"type\":\"domain_control_validation_failed\",\"details\":{\"turnchat.pancake.vn\":{\"http:\\/\\/turnchat.pancake.vn\\/.well-known\\/pki-validation\\/EEB6CD0CAA256FB76E4BCB825059E916.txt\":{\"file_found\":0,\"error\":true,\"error_slug\":\"bad_response_code\",\"error_info\":\"Server responded with status code: 404 (200 expected)\"}}}}} decode_error=json: unknown field \"success\")"}
{"level":"error","ts":1717412945.828925,"logger":"tls.obtain","msg":"will retry","error":"[turnchat.pancake.vn] Obtain: verifying identifiers: POST https://api.zerossl.com/certificates/bd60abc3077f80f689d93b32d315023d/challenges?access_key=redacted: HTTP 200: API error 0: domain_control_validation_failed (details=map[turnchat.pancake.vn:map[http://turnchat.pancake.vn/.well-known/pki-validation/EEB6CD0CAA256FB76E4BCB825059E916.txt:{{0 0   } {0 true bad_response_code Server responded with status code: 404 (200 expected)}}]]) (raw={\"success\":false,\"error\":{\"code\":0,\"type\":\"domain_control_validation_failed\",\"details\":{\"turnchat.pancake.vn\":{\"http:\\/\\/turnchat.pancake.vn\\/.well-known\\/pki-validation\\/EEB6CD0CAA256FB76E4BCB825059E916.txt\":{\"file_found\":0,\"error\":true,\"error_slug\":\"bad_response_code\",\"error_info\":\"Server responded with status code: 404 (200 expected)\"}}}}} decode_error=json: unknown field \"success\")","attempt":5,"retrying_in":600,"elapsed":619.307460524,"max_duration":2592000}
{"level":"info","ts":1717412946.8676581,"logger":"tls.issuance.zerossl","msg":"canceled certificate","identifiers":["video.pancake.vn"],"cert_id":"17423b951963093e86c274f8483f9af5","verification_method":"HTTP_CSR_HASH"}
{"level":"error","ts":1717412946.8677192,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"video.pancake.vn","issuer":"zerossl","error":"verifying identifiers: POST https://api.zerossl.com/certificates/17423b951963093e86c274f8483f9af5/challenges?access_key=redacted: HTTP 200: API error 0: domain_control_validation_failed (details=map[video.pancake.vn:map[http://video.pancake.vn/.well-known/pki-validation/2F5B7B85C16297392C6803FA6C94BE35.txt:{{0 0   } {0 true bad_response_code Server responded with status code: 404 (200 expected)}}]]) (raw={\"success\":false,\"error\":{\"code\":0,\"type\":\"domain_control_validation_failed\",\"details\":{\"video.pancake.vn\":{\"http:\\/\\/video.pancake.vn\\/.well-known\\/pki-validation\\/2F5B7B85C16297392C6803FA6C94BE35.txt\":{\"file_found\":0,\"error\":true,\"error_slug\":\"bad_response_code\",\"error_info\":\"Server responded with status code: 404 (200 expected)\"}}}}} decode_error=json: unknown field \"success\")"}
{"level":"error","ts":1717412946.8678486,"logger":"tls.obtain","msg":"will retry","error":"[video.pancake.vn] Obtain: verifying identifiers: POST https://api.zerossl.com/certificates/17423b951963093e86c274f8483f9af5/challenges?access_key=redacted: HTTP 200: API error 0: domain_control_validation_failed (details=map[video.pancake.vn:map[http://video.pancake.vn/.well-known/pki-validation/2F5B7B85C16297392C6803FA6C94BE35.txt:{{0 0   } {0 true bad_response_code Server responded with status code: 404 (200 expected)}}]]) (raw={\"success\":false,\"error\":{\"code\":0,\"type\":\"domain_control_validation_failed\",\"details\":{\"video.pancake.vn\":{\"http:\\/\\/video.pancake.vn\\/.well-known\\/pki-validation\\/2F5B7B85C16297392C6803FA6C94BE35.txt\":{\"file_found\":0,\"error\":true,\"error_slug\":\"bad_response_code\",\"error_info\":\"Server responded with status code: 404 (200 expected)\"}}}}} decode_error=json: unknown field \"success\")","attempt":5,"retrying_in":600,"elapsed":620.346353915,"max_duration":2592000}

I had double check my api key, I have set up and issued cert on ZeroSSL, but why Caddy still create for me and cancel it?

Thanks!

1 Like

I am getting the same error on 2.8 …not sure what is the issue . Using caddy with redis storage extension . Same config is working exactly fine on old version

This might be a bug. I’ll look into it.

1 Like

Update: I’m not able to reproduce the behavior. “It works on my machine” (sorry, I know that sucks to hear!)

To clarify, 2.8.1 can get certs, 2.8.4 can’t??

Can I please get everyone’s exact configs and exact Caddy versions?

If you’re using < 2.8, please upgrade to 2.8.4 first.

1 Like

Thanks for your replies!

Can I please get everyone’s exact configs and exact Caddy versions?

My caddy version is 2.8.1 and my config is in my first post. I cannot get certs.

I only wander why caddy create new zeroSSL certs for me many times (I have posted my logs in above comment). Do I make any wrong configurations?

Can you change the log level to DEBUG and upgrade to 2.8.4 and try again? If it still doesn’t work then I’ll be able to dig into it more.

1 Like

Yes, I have updated my Caddy Image to v2.8.4. Below are my logs:

{"level":"info","ts":1717987549.1015604,"msg":"using config from file","file":"/etc/caddy.yaml"}
{"level":"info","ts":1717987549.1028702,"msg":"adapted config to JSON","adapter":"yaml"}
{"level":"info","ts":1717987549.1048381,"logger":"admin","msg":"admin endpoint started","address":"localhost:2019","enforce_origin":false,"origins":["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]}
{"level":"info","ts":1717987549.1059127,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc000486800"}
{"level":"debug","ts":1717987549.106823,"logger":"layer4","msg":"listening","address":"tcp/[::]:4433"}
{"level":"info","ts":1717987549.107203,"logger":"tls.obtain","msg":"acquiring lock","identifier":"video.pancake.vn"}
{"level":"info","ts":1717987549.1072254,"logger":"tls.obtain","msg":"acquiring lock","identifier":"turnchat.pancake.vn"}
{"level":"info","ts":1717987549.1078694,"msg":"autosaved config (load with --resume flag)","file":"/root/.config/caddy/autosave.json"}
{"level":"info","ts":1717987549.107898,"msg":"serving initial configuration"}
{"level":"info","ts":1717987549.1140442,"logger":"tls.obtain","msg":"lock acquired","identifier":"video.pancake.vn"}
{"level":"info","ts":1717987549.114057,"logger":"tls.obtain","msg":"lock acquired","identifier":"turnchat.pancake.vn"}
{"level":"info","ts":1717987549.1141596,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"video.pancake.vn"}
{"level":"info","ts":1717987549.1142664,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"turnchat.pancake.vn"}
{"level":"info","ts":1717987549.1142826,"logger":"tls","msg":"storage cleaning happened too recently; skipping for now","storage":"FileStorage:/data","instance":"afce6d53-41cc-4a72-ba5b-10f3ed71a40b","try_again":1718073949.1142788,"try_again_in":86399.999999311}
{"level":"debug","ts":1717987549.1144187,"logger":"events","msg":"event","name":"cert_obtaining","id":"a311dfcf-4fe1-407f-b1bd-593b1a525626","origin":"tls","data":{"identifier":"turnchat.pancake.vn"}}
{"level":"debug","ts":1717987549.1142774,"logger":"events","msg":"event","name":"cert_obtaining","id":"9762f0ce-e9f7-407a-9bdf-ba1e27ab5379","origin":"tls","data":{"identifier":"video.pancake.vn"}}
{"level":"info","ts":1717987549.1145515,"logger":"tls","msg":"finished cleaning storage units"}
{"level":"debug","ts":1717987549.1149166,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"zerossl"}
{"level":"debug","ts":1717987549.1149137,"logger":"tls.obtain","msg":"trying issuer 1/1","issuer":"zerossl"}
{"level":"info","ts":1717987549.1151009,"logger":"tls.issuance.zerossl","msg":"creating certificate","identifiers":["turnchat.pancake.vn"]}
{"level":"info","ts":1717987549.115895,"logger":"tls.issuance.zerossl","msg":"creating certificate","identifiers":["video.pancake.vn"]}
{"level":"error","ts":1717987550.2781017,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"video.pancake.vn","issuer":"zerossl","error":"creating certificate: POST https://api.zerossl.com/certificates?access_key=redacted: HTTP 200: API error 2817: certificate_limit_reached (details=map[]) (raw={\"success\":false,\"error\":{\"code\":2817,\"type\":\"certificate_limit_reached\"}} decode_error=json: unknown field \"success\")"}
{"level":"debug","ts":1717987550.278267,"logger":"events","msg":"event","name":"cert_failed","id":"30fc42a4-7736-4cdc-8500-74db8d0b9c77","origin":"tls","data":{"error":{},"identifier":"video.pancake.vn","issuers":["zerossl"],"renewal":false}}
{"level":"error","ts":1717987550.2782965,"logger":"tls.obtain","msg":"will retry","error":"[video.pancake.vn] Obtain: creating certificate: POST https://api.zerossl.com/certificates?access_key=redacted: HTTP 200: API error 2817: certificate_limit_reached (details=map[]) (raw={\"success\":false,\"error\":{\"code\":2817,\"type\":\"certificate_limit_reached\"}} decode_error=json: unknown field \"success\")","attempt":1,"retrying_in":60,"elapsed":1.164224709,"max_duration":2592000}
{"level":"error","ts":1717987550.3299913,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"turnchat.pancake.vn","issuer":"zerossl","error":"creating certificate: POST https://api.zerossl.com/certificates?access_key=redacted: HTTP 200: API error 2817: certificate_limit_reached (details=map[]) (raw={\"success\":false,\"error\":{\"code\":2817,\"type\":\"certificate_limit_reached\"}} decode_error=json: unknown field \"success\")"}
{"level":"debug","ts":1717987550.3301113,"logger":"events","msg":"event","name":"cert_failed","id":"849579d1-4690-4bf4-8255-24a577ab76a7","origin":"tls","data":{"error":{},"identifier":"turnchat.pancake.vn","issuers":["zerossl"],"renewal":false}}
{"level":"error","ts":1717987550.3301692,"logger":"tls.obtain","msg":"will retry","error":"[turnchat.pancake.vn] Obtain: creating certificate: POST https://api.zerossl.com/certificates?access_key=redacted: HTTP 200: API error 2817: certificate_limit_reached (details=map[]) (raw={\"success\":false,\"error\":{\"code\":2817,\"type\":\"certificate_limit_reached\"}} decode_error=json: unknown field \"success\")","attempt":1,"retrying_in":60,"elapsed":1.216062079,"max_duration":2592000}

It seems my server has requested many times to ZeroSSL, but I don’t know why.

Thanks!

What kind of plan do you have for your account? Free plans can only have a few certificates from their API.

1 Like

Yes, I use free plan. I issued only 1 cert on ZeroSSL page.

If your account is only showing 1 certificate issued, then you may have to contact their support to figure out why, since the free plan should allow 3.

1 Like

Thanks, I mean, why I issued 1 certificate and caddy still generate for me, when I run Caddy with Livekit. And I have another question, if I have another SSL cert, can I add to Caddy configuration?
Thank you!

If the certificate is in its storage in the expected place, Caddy won’t know about it and will have to generate a cert.

To use an existing cert, use the tls directive: tls (Caddyfile directive) — Caddy Documentation

1 Like