Issuing certificates for third-party (custom) domains

[jmstfv (Jama)]

1. My Caddy version (caddy version):

v2.0.0-beta.14 h1:QX1hRMfTA5sel53o5SuON1ys50at6yuSAnPr56sLeK8=

Built with a tls.dns.cloudflare module (see this post):

$ caddy list-modules | grep "tls"
tls
tls.certificate_selection.custom
tls.certificates.automate
tls.certificates.load_files
tls.certificates.load_folders
tls.certificates.load_pem
tls.dns.cloudflare
tls.handshake_match.sni
tls.management.acme
tls.stek.distributed
tls.stek.standard

2. How I run Caddy:

a. System environment:

• Ubuntu 18.04.4 (kernel: 4.15.0-1060-aws)
• systemd 237

b. Command:

After copying all config files and creating caddy user/group:

sudo systemctl daemon-reload
sudo systemctl enable caddy
sudo systemctl start caddy

c. Service/unit/compose file:

Vanilla systemd config for Caddy:

[Unit]
Description=Caddy Web Server
Documentation=https://caddyserver.com/docs/
After=network.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --config /etc/caddy/caddy.json --resume --environ
ExecReload=/usr/bin/caddy reload --config /etc/caddy/caddy.json
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

{
  "apps": {
    "tls": {
      "automation": {
        "policies": [{
          "management": {
            "module": "acme",
            "email": "MY CLOUDFLARE EMAIL",
            "challenges": {
              "dns": {
                "provider": "cloudflare",
                "api_token": "MY API TOKEN"
              }
            }
          }
        }]
      }
    },
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "*.jmstfv.com",
                    "status.tryhexadecimal.com"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "encodings": {
                            "gzip": {},
                            "zstd": {}
                          },
                          "handler": "encode"
                        },
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "set": {
                                "Host": [
                                  "{http.request.host}"
                                ],
                                "X-Forwarded-For": [
                                  "{http.request.remote.host}"
                                ],
                                "X-Forwarded-Port": [
                                  "{server_port}"
                                ],
                                "X-Forwarded-Proto": [
                                  "{http.request.scheme}"
                                ],
                                "X-Real-Ip": [
                                  "{http.request.remote.host}"
                                ]
                              }
                            }
                          },
                          "upstreams": [
                            {
                              "dial": "0.0.0.0:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

3. The problem I’m having:

I can’t obtain a certificate for status.tryhexadecimal.com. My guess:

• incorrect permissions on the Cloudflare API token (see below)
• misconfiguration of DNS records (see below)

Wildcard certificate for *.jmstfv.com works fine.

4. Error messages and/or full log output:

2020/02/25 08:29:21 [INFO][status.tryhexadecimal.com] Obtain certificate
2020/02/25 08:29:21 [INFO][status.tryhexadecimal.com] Obtain: Waiting on rate limiter...
2020/02/25 08:29:21 [INFO][status.tryhexadecimal.com] Obtain: Done waiting
2020/02/25 08:29:21 [INFO] [status.tryhexadecimal.com] acme: Obtaining bundled SAN certificate
2020/02/25 08:29:21 [INFO] nonce error retry: acme: error: 400 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:badNonce :: JWS has an invalid anti-replay nonce: "00021MzqJMAx36hZQ5FTBnaK1IVkSodW-GqWcMiOxF-8rLI", url:
2020/02/25 08:29:22 [INFO] [status.tryhexadecimal.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3001949923
2020/02/25 08:29:22 [INFO] [status.tryhexadecimal.com] acme: Could not find solver for: tls-alpn-01
2020/02/25 08:29:22 [INFO] [status.tryhexadecimal.com] acme: Could not find solver for: http-01
2020/02/25 08:29:22 [INFO] [status.tryhexadecimal.com] acme: use dns-01 solver
2020/02/25 08:29:22 [INFO] [status.tryhexadecimal.com] acme: Preparing to solve DNS-01
2020/02/25 08:29:22 [INFO] [status.tryhexadecimal.com] acme: Cleaning DNS-01 challenge
2020/02/25 08:29:22 [WARN] [status.tryhexadecimal.com] acme: error cleaning up: cloudflare: failed to find zone tryhexadecimal.com.: ListZonesContext command failed: error from makeRequest: HTTP status 403: insufficient permissions
2020/02/25 08:29:22 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3001949923
2020/02/25 08:29:22 [ERROR][status.tryhexadecimal.com] failed to obtain certificate: acme: Error -> One or more domains had a problem:
[status.tryhexadecimal.com] [status.tryhexadecimal.com] acme: error presenting token: cloudflare: failed to find zone tryhexadecimal.com.: ListZonesContext command failed: error from makeRequest: HTTP status 403: insufficient permissions (attempt 1/2; challenge=dns-01)
2020/02/25 08:29:23 [INFO] [status.tryhexadecimal.com] acme: Obtaining bundled SAN certificate
2020/02/25 08:29:24 [INFO] [status.tryhexadecimal.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3001950343
2020/02/25 08:29:24 [INFO] [status.tryhexadecimal.com] acme: Could not find solver for: tls-alpn-01
2020/02/25 08:29:24 [INFO] [status.tryhexadecimal.com] acme: Could not find solver for: http-01
2020/02/25 08:29:24 [INFO] [status.tryhexadecimal.com] acme: use dns-01 solver
2020/02/25 08:29:24 [INFO] [status.tryhexadecimal.com] acme: Preparing to solve DNS-01
2020/02/25 08:29:24 [INFO] [status.tryhexadecimal.com] acme: Cleaning DNS-01 challenge
2020/02/25 08:29:24 [WARN] [status.tryhexadecimal.com] acme: error cleaning up: cloudflare: failed to find zone tryhexadecimal.com.: ListZonesContext command failed: error from makeRequest: HTTP status 403: insufficient permissions
2020/02/25 08:29:24 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3001950343
2020/02/25 08:29:24 [ERROR][status.tryhexadecimal.com] failed to obtain certificate: acme: Error -> One or more domains had a problem:
[status.tryhexadecimal.com] [status.tryhexadecimal.com] acme: error presenting token: cloudflare: failed to find zone tryhexadecimal.com.: ListZonesContext command failed: error from makeRequest: HTTP status 403: insufficient permissions (attempt 2/2; challenge=dns-01)
2020/02/25 08:29:25 [ERROR] status.tryhexadecimal.com: obtaining certificate: failed to obtain certificate: acme: Error -> One or more domains had a problem:
[status.tryhexadecimal.com] [status.tryhexadecimal.com] acme: error presenting token: cloudflare: failed to find zone tryhexadecimal.com.: ListZonesContext command failed: error from makeRequest: HTTP status 403: insufficient permissions - backing off and retrying (attempt 9/46)...

Sending a request to status.tryhexadecimal.com using curl:

$ curl -v https://status.tryhexadecimal.com
[...]
curl: (35) error:14094438:SSL routines:ssl3_read_bytes:tlsv1 alert internal error

Caddy logs:

http: TLS handshake error from [IP:Port]: no certificate available for 'status.tryhexadecimal.com'

5. What I already tried:

DNS A record for jmstfv.com (wildcard DNS record):

$ nslookup *.jmstfv.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
Name:	*.jmstfv.com
Address: 34.229.26.68

DNS CNAME record for status.tryhexadecimal.com (note: there isn’t a separate DNS record for domains.jmstfv.com, it resolves to *.jmstfv.com):

$ nslookup status.tryhexadecimal.com
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
status.tryhexadecimal.com	canonical name = domains.jmstfv.com.
Name:	domains.jmstfv.com
Address: 34.229.26.68

DNS CAA record for tryhexadecimal.com:

$ dig tryhexadecimal.com type257 +short
0 issuewild "comodoca.com"
0 issuewild "digicert.com"
0 issuewild "letsencrypt.org"
0 issue "comodoca.com"
0 issue "digicert.com"
0 issue "letsencrypt.org"

DNS CAA record for status.tryhexadecimal.com:

$ dig status.tryhexadecimal.com type257 +short
domains.jmstfv.com.

Permissions on the Cloudflare token (API token belongs to the jmstfv.com account):

Zone / Zone / Read
Zone / DNS / Edit

Tried adding Zone / Zone Settings / Edit to no avail. Also tried setting above permissions to Edit but that didn’t work either.

6. Links to relevant resources:

Similar error messages (for traefik):

• https://www.reddit.com/r/kubernetes/comments/ebfh5m/kubernetes_and_traefik_with_letsencrypt/
• Cert Provider Cloudflare - CF_DNS_API_TOKEN integration issue · Issue #5965 · traefik/traefik · GitHub
• Api token for traefik dns challenge? - API - Cloudflare Community

[Whitestrake (Matthew Fay)]

@jmstfv

2020/02/25 08:29:22 [WARN] [status.tryhexadecimal.com] acme: error cleaning up: cloudflare: failed to find zone tryhexadecimal.com.: ListZonesContext command failed: error from makeRequest: HTTP status 403: insufficient permissions

Yep, it’s a permissions issue alright. Cloudflare is rejecting your attempt to read the zone.

Don’t know why though. Zone / Zone / Read and Zone / DNS / Edit should work according to a recent post:

Caddy 2: wildcard + on demand certificates

For future visitors: API permissions you need to give (from your Cloudflare dashboard) so that Caddy could request a wildcard certificate [cloudflare-dashboard]

Just to test - does it work with any other domains? Does it work with a global API key instead of a scoped API key?

[jmstfv (Jama)]

@Whitestrake

Don’t know why though. Zone / Zone / Read and Zone / DNS / Edit should work according to a recent post:

Those permissions work if you’re getting a certificate for your own domain. But the thing is that those domains are in separate Cloudflare accounts.

@Whitestrake

Does it work with a global API key instead of a scoped API key?

Weirdly enough, Caddy throws 400 when using a Global API token:

HTTP status 400: content "{\"success\":false,\"errors\":[{\"code\":6003,\"message\":\"Invalid request headers\",\"error_chain\":[{\"code\":6111,\"message\":\"Invalid format for Authorization header\"}]}],\"messages\":[],\"result\":null}" (attempt 2/2; challenge=dns-01)

@Whitestrake

Just to test - does it work with any other domains?

I purchased a different domain, from a different registrar, with different NS records, but the problem still persists. It seems like Cloudflare tries to read the zone of the third-party domain, but since the domain in questions doesn’t reside in that Cloudflare account, it throws 403

[jmstfv (Jama)]

Progress report: to debug the problem, I decided to dive one level lower. I tried to pass a DNS challenge with lego and see what happens (note that tryhexadecimal.com is hosted on a separate Cloudflare account from jmstfv.com):

CLOUDFLARE_EMAIL="cloudflare@jmstfv.com" \
CLOUDFLARE_API_KEY="GLOBAL API TOKEN" \
go/bin/lego --dns cloudflare --domains status.tryhexadecimal.com --email cloudflare@jmstfv.com --dns.resolvers 8.8.8.8 run

Here’s the debug log:

2020/02/27 18:12:06 [INFO] [status.tryhexadecimal.com] acme: Obtaining bundled SAN certificate
2020/02/27 18:12:07 [INFO] [status.tryhexadecimal.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3044936973
2020/02/27 18:12:07 [INFO] [status.tryhexadecimal.com] acme: Could not find solver for: tls-alpn-01
2020/02/27 18:12:07 [INFO] [status.tryhexadecimal.com] acme: Could not find solver for: http-01
2020/02/27 18:12:07 [INFO] [status.tryhexadecimal.com] acme: use dns-01 solver
2020/02/27 18:12:07 [INFO] [status.tryhexadecimal.com] acme: Preparing to solve DNS-01

2020/02/27 18:12:07 [INFO] [status.tryhexadecimal.com] acme: Cleaning DNS-01 challenge
2020/02/27 18:12:07 [WARN] [status.tryhexadecimal.com] acme: error cleaning up: cloudflare: failed to find zone tryhexadecimal.com.: Zone could not be found
2020/02/27 18:12:07 [INFO] Deactivating auth: https://acme-v02.api.letsencrypt.org/acme/authz-v3/3044936973
2020/02/27 18:12:08 Could not obtain certificates:
	acme: Error -> One or more domains had a problem:
[status.tryhexadecimal.com] [status.tryhexadecimal.com] acme: error presenting token: cloudflare: failed to find zone tryhexadecimal.com.: Zone could not be found

I went searching, and landed on the following Github issue (the linked comment explains the problem):

The Cloudflare error happens, because in order to update DNS records for the domain you’re trying to obtain certificates for, Lego needs to find the “apex name” for that domain. This is the domain, for which a SOA records exists.

So I ran the dig query for status.tryhexadecimal.com:

$ dig -t SOA status.tryhexadecimal.com +recurse +nocomment

; <<>> DiG 9.11.3-1ubuntu1.11-Ubuntu <<>> -t SOA status.tryhexadecimal.com +recurse +nocomment
;; global options: +cmd
;status.tryhexadecimal.com.	IN	SOA
status.tryhexadecimal.com. 60	IN	CNAME	domains.jmstfv.com.
;; Query time: 8 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Thu Feb 27 18:14:52 UTC 2020
;; MSG SIZE  rcvd: 83

Since the Cloudflare account in question doesn’t host the tryhexadecimal.com domain, lego can’t pass the DNS challenge, thus Caddy can’t issue a certificate.

So. I commented out the dns block from caddy.json, and managed to obtain a certificate for status.tryhexadecimal.com

TL;DR: DNS challenge works for wildcard certificates, HTTP or TLS-ALPN challenge works for third-party domains. If you’re using a DNS challenge, others are disabled by default. (source).

Just a sanity check. Is the following configuration correct?

"challenges": {
  "http": { "disabled": false },
  "tls-alpn": { "disabled": false },
  "dns": {
    "provider": "cloudflare",
    "api_token": "MY API TOKEN"
  }
}

[Whitestrake (Matthew Fay)]

I believe so! Should be all you need.

[matt (Matt Holt)]

So, is this resolved?

Edit: Why does your config work? Usually, the DNS challenge is used when the HTTP/TLS-ALPN challenges cannot be. So why does using all of them work for you?

[jmstfv (Jama)]

@matt

So, is this resolved?

Not quite sure about it. I have tried running the above config one more time, and it failed: when DNS challenge is present, HTTP/TLS-ALPN doesn’t get used, even if I explicitly pass disabled: false line to both http and tls-alpn blocks. Is this the expected behavior when passing disabled: false line to both of these blocks (given that dns block is present)?

@matt

Why does your config work? Usually, the DNS challenge is used when the HTTP/TLS-ALPN challenges cannot be. So why does using all of them work for you?

I should have mentioned in my original post that my use case is quite typical among SaaS applications that provide websites for their customers (e.g. static site hosting, status pages, podcast websites, newsletter signup pages, etc…). I need to issue both a wildcard certificate for my own domain (to serve customers’ websites from a subdomain) and certificates for third-party domains (so that customers can serve their websites from their own domain). So in a sense, I need both of these challenges to work side-by-side.

One workaround around this would be to use Caddy’s API to remove the dns block from config every time I want to issue a certificate for a third-party domain (so that Caddy could use HTTP/TLS-ALPN challenge), and then when everything is done, adding the dns block back

[matt (Matt Holt)]

@jmstfv

Is this the expected behavior when passing disabled: false line to both of these blocks (given that dns block is present)?

Yeah, enabling the DNS challenge disables the others, because the DNS challenge is necessarily used when the others can’t be. It’s too error-prone to expect users to always disable the other two challenge types when they enable DNS.

I still don’t get why you need all 3 challenges for the same domain names.

@jmstfv

I need to issue both a wildcard certificate for my own domain (to serve customers’ websites from a subdomain) and certificates for third-party domains (so that customers can serve their websites from their own domain).

So, are the SANs on the certificates different or the same? Please illustrate exactly what you are trying to accomplish… that will help us understand what needs to be done.

[jmstfv (Jama)]

Posting my working JSON config as a reference (note: this config only works with beta v15).

{
  "apps": {
    "tls": {
      "automation": {
        "policies": [{
          "hosts": ["*.tryhexadecimal.com"],
          "management": {
            "module": "acme",
            "email": "YOUR EMAIL HERE",
            "challenges": {
              "dns": {
                "provider": "cloudflare",
                "api_token": "YOUR API TOKEN HERE"
              }
            }
          }
        },
        {
          "management": {
            "module": "acme",
            "email": "YOUR EMAIL HERE",
            "on_demand": true
          }
        }],
        "on_demand": {
          "ask": "YOUR WEBHOOK URL"
        }
      }
    },
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            ":443"
          ],
          "routes": [
            {
              "match": [
                {
                  "host": [
                    "*.tryhexadecimal.com",
                    "*.*.*"
                  ]
                }
              ],
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "encodings": {
                            "gzip": {}
                          },
                          "handler": "encode"
                        },
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "set": {
                                "Host": [
                                  "{http.request.host}"
                                ],
                                "X-Forwarded-For": [
                                  "{http.request.remote.host}"
                                ],
                                "X-Forwarded-Port": [
                                  "{server_port}"
                                ],
                                "X-Forwarded-Proto": [
                                  "{http.request.scheme}"
                                ],
                                "X-Real-Ip": [
                                  "{http.request.remote.host}"
                                ]
                              }
                            }
                          },
                          "upstreams": [
                            {
                              "dial": "0.0.0.0:3000"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

By default, you won’t’ be able to use HTTP/TLS-ALPN challenges with a DNS challenge. The way around it is to scope your policies by hosts (i.e. using hosts directive).

Thanks @Whitestrake and @matt for your help!

[Sebastian_Perez]

I cant find the “hosts” nor “management” directive under policies in the docs : https://caddyserver.com/docs/json/apps/tls/automation/policies/

Should it work with “subjects” and “issuer” instead ?

[francislavoie]

That’s right @Sebastian_Perez. This thread was from Caddy v2 beta 14, and the names of those changed before v2 stable was released.