Linode DNS for Let's Encrypt

denverpedro · May 6, 2024, 9:13pm

1. The problem I’m having:

I have tried both Linode add-ons for Caddy and with both I am receiving errors when I try to utilize DNS validation for Let’s Encrypt certificates. I must be doing something wrong, but I can’t figure it out. If anyone has used Linode for DNS validation with Caddy/Let’s Encrypt, please help!

2. Error messages and/or full log output:

My systemctl logs look like this:

May 06 20:24:20 morpheus caddy[2556205]: {"level":"info","ts":1715027060.766635,"logger":"tls.obtain","msg":"obtaining certificate","identifier":"test2.beersmas.com"}
May 06 20:24:20 morpheus caddy[2556205]: {"level":"info","ts":1715027060.7674685,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["test2.beersmas.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
May 06 20:24:20 morpheus caddy[2556205]: {"level":"info","ts":1715027060.767484,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["test2.beersmas.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
May 06 20:24:21 morpheus caddy[2556205]: {"level":"info","ts":1715027061.0487003,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"test2.beersmas.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
May 06 20:24:21 morpheus caddy[2556205]: {"level":"error","ts":1715027061.3639472,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"test2.beersmas.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.test2.beersmas.com\" (usually OK if presenting also failed)"}
May 06 20:24:21 morpheus caddy[2556205]: {"level":"error","ts":1715027061.3862052,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test2.beersmas.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[test2.beersmas.com] solving challenges: presenting for challenge: adding temporary record for zone \"beersmas.com.\": could not find domain ID for zone: beersmas.com.: could not list domains: [401] Invalid Token (order=https://acme-v02.api.letsencrypt.org/acme/order/1712058617/267163312347) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
May 06 20:24:21 morpheus caddy[2556205]: {"level":"info","ts":1715027061.3869252,"logger":"tls.issuance.zerossl","msg":"waiting on internal rate limiter","identifiers":["test2.beersmas.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
May 06 20:24:21 morpheus caddy[2556205]: {"level":"info","ts":1715027061.3869581,"logger":"tls.issuance.zerossl","msg":"done waiting on internal rate limiter","identifiers":["test2.beersmas.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":"caddy@zerossl.com"}
May 06 20:24:22 morpheus caddy[2556205]: {"level":"info","ts":1715027062.0235941,"logger":"tls.issuance.zerossl.acme_client","msg":"trying to solve challenge","identifier":"test2.beersmas.com","challenge_type":"dns-01","ca":"https://acme.zerossl.com/v2/DV90"}
May 06 20:24:22 morpheus caddy[2556205]: {"level":"error","ts":1715027062.0909102,"logger":"tls.issuance.zerossl.acme_client","msg":"cleaning up solver","identifier":"test2.beersmas.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.test2.beersmas.com\" (usually OK if presenting also failed)"}
May 06 20:24:22 morpheus caddy[2556205]: {"level":"error","ts":1715027062.190231,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"test2.beersmas.com","issuer":"acme.zerossl.com-v2-DV90","error":"[test2.beersmas.com] solving challenges: presenting for challenge: adding temporary record for zone \"beersmas.com.\": could not find domain ID for zone: beersmas.com.: could not list domains: [401] Invalid Token (order=https://acme.zerossl.com/v2/DV90/order/BX6cBdA_0ieJaqy0G5EYdQ) (ca=https://acme.zerossl.com/v2/DV90)"}
May 06 20:24:22 morpheus caddy[2556205]: {"level":"error","ts":1715027062.1903296,"logger":"tls.obtain","msg":"will retry","error":"[test2.beersmas.com] Obtain: [test2.beersmas.com] solving challenges: presenting for challenge: adding temporary record for zone \"beersmas.com.\": could not find domain ID for zone: beersmas.com.: could not list domains: [401] Invalid Token (order=https://acme.zerossl.com/v2/DV90/order/BX6cBdA_0ieJaqy0G5EYdQ) (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":1.423860267,"max_duration":2592000}

And if I chase the URLs in the log lines I see the following:

{
  "status": "invalid",
  "expires": "2024-05-13T20:24:20Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "test2.beersmas.com"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/347513587217"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/1712058617/267163312347"
}

And this:

{"type":"urn:ietf:params:acme:error:malformed","status":405,"detail":"The request message was malformed"}

And “Authorizations” request from the first link:

{
  "identifier": {
    "type": "dns",
    "value": "test2.beersmas.com"
  },
  "status": "deactivated",
  "expires": "2024-05-13T20:24:20Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/347513587217/DG2c7A",
      "token": "CNvD2mmgkUCjFhZitF5rilKp4ZO87LFfsPXa_ACvziI"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/347513587217/GpboQQ",
      "token": "CNvD2mmgkUCjFhZitF5rilKp4ZO87LFfsPXa_ACvziI"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/347513587217/M032xg",
      "token": "CNvD2mmgkUCjFhZitF5rilKp4ZO87LFfsPXa_ACvziI"
    }
  ]
}

And “finalize” link from the first link:

{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Method not allowed",
  "status": 405
}

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

I downloaded a binary from the website. Specifically, I selected the base options + the dns.providers.linode add-on (I have tried BOTH the tosie/caddy-dns-linode add-on and the caddy-dns/linode add on and both with the same results.

I then installed that in /usr/local/bin/caddy and ran caddy as a systemctl service.

a. System environment:

My system is running Ubuntu 20.04 (Ubuntu 22.04.4 LTS). Caddy is NOT running in docker it is standalone as described above.

b. Command:

sudo systemctl enable --now caddy

c. Service/unit/compose file:

Not relevant.

d. My complete Caddy config:

{
        acme_dns linode {"REMOVED TOKEN"}
}

(trusted_proxy_list) {
       trusted_proxies 10.0.1.0/24
}

## Unprotected Endpoints (no forced auth)
ha.beersmas.com {
        reverse_proxy 10.0.1.30:8123 {
                import trusted_proxy_list
        }
}
nas.beersmas.com {
        reverse_proxy 10.0.1.20:5000 {
                import trusted_proxy_list
        }
}
auth.beersmas.com {
        reverse_proxy 10.0.1.40:9091 {
                import trusted_proxy_list
        }
}

## Protected Endpoints (force auth)
radarr.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:7878 {
                import trusted_proxy_list
        }
}
lidarr.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:8686 {
                import trusted_proxy_list
        }
}
sonarr.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:8989 {
                import trusted_proxy_list
        }
}
sabnzbd.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:8080 {
                import trusted_proxy_list
        }
}

ombi.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:3579 {
                import trusted_proxy_list
        }
}
vs.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:8443 {
                import trusted_proxy_list
        }
}
test2.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:8443 {
                import trusted_proxy_list
	}
}

5. Links to relevant resources:

Bruce5051 · May 6, 2024, 9:28pm

Hello @denverpedro,

I don’t think the domain name test2.beersmas.com exists.

Using the online tool https://unboundtest.com/ yields these results
QUERY, status: NXDOMAIN when just looking for the CAA.
https://unboundtest.com/m/CAA/test2.beersmas.com/HHROO3G2

Query results for CAA test2.beersmas.com

Response:
;; opcode: QUERY, status: NXDOMAIN, id: 14447
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 512

;; QUESTION SECTION:
;test2.beersmas.com.	IN	 CAA

;; AUTHORITY SECTION:
beersmas.com.	0	IN	SOA	ns1.linode.com. denver\.pete.gmail.com. 2021000140 14400 14400 1209600 86400

----- Unbound logs -----
May 06 21:25:21 unbound1.19[2030642:0] debug: creating udp6 socket ::1 1053
May 06 21:25:21 unbound1.19[2030642:0] debug: creating tcp6 socket ::1 1053

And using nslookup I get similar results

$ nslookup test2.beersmas.com ns1.linode.com.
Server:         ns1.linode.com.
Address:        92.123.94.2#53

** server can't find test2.beersmas.com: NXDOMAIN

In contrast to successfully finding www.beersmas.com

$ nslookup www.beersmas.com ns1.linode.com.
Server:         ns1.linode.com.
Address:        92.123.94.2#53

Name:   www.beersmas.com
Address: 45.56.127.43
Name:   www.beersmas.com
Address: 2600:3c00::f03c:93ff:fef4:d475

$ nslookup -q=all www.beersmas.com ns1.linode.com.
unknown query type: all
Server:         ns1.linode.com.
Address:        92.123.94.2#53

Name:   www.beersmas.com
Address: 45.56.127.43
Name:   www.beersmas.com
Address: 2600:3c00::f03c:93ff:fef4:d475

$ nslookup -q=all _acme-challenge.test2.beersmas.com ns1.linode.com.
unknown query type: all
Server:         ns1.linode.com.
Address:        92.123.94.2#53

** server can't find _acme-challenge.test2.beersmas.com: NXDOMAIN

$ nslookup -q=all test2.beersmas.com ns1.linode.com.
unknown query type: all
Server:         ns1.linode.com.
Address:        92.123.94.2#53

** server can't find test2.beersmas.com: NXDOMAIN

denverpedro · May 6, 2024, 9:54pm

Thank you Bruce for the quick response. You are correct, I’ve been doing so much troubleshooting that I’ve added/removed too many variables. I re-added the DNS record for test2.beersmas.com and receive the same error.

I have certificates that have been successfully created using the HTTP/HTTPS validation method, and if I remove my cert stores for Caddy and re-start it I get the same error for ALL my hostnames. So it doesn’t have anything to do with the actual hostname at this point (but I did re-create test2 just in case).

What I mean is that I set this up initially with HTTP/HTTPS validation, that worked just fine. But I would like to move to a wildcard cert so I want to use DNS validation. I set up the DNS validation, and added a new host (test2 in this case) to see if it will generate the new cert and this is the error I run into. As mentioned above, in my troubleshooting I even attempted to delete all of the data/stores for Caddy and re-start the cert request process. That failed with DNS, I had to fall back to HTTP/HTTPS to get the certs re-created so I can continue to use my services.

Any other suggestions?

Bruce5051 · May 6, 2024, 10:10pm

Yes; I see the CNAME to home.beersmas.com

Not at the moment, sorry; I am late for an appointment.

Bruce5051 · May 7, 2024, 3:16am

denverpedro:

vs.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:8443 {
                import trusted_proxy_list
        }
}
test2.beersmas.com {
        forward_auth 10.0.1.40:9091 {
                uri /api/verify?rd=https://auth.beersmas.com:8001
                copy_headers Remote-User Remote-Groups Remote-Email Remote-Name
                import trusted_proxy_list
        }
        reverse_proxy 10.0.1.40:8443 {
                import trusted_proxy_list
	}
}

Maybe I am missing something here; but other than name those 2 look identical to me. Am I seeing that correctly?

denverpedro · May 7, 2024, 3:36am

Yep, I appreciate the second pair of eyes but I literally duplicated that to just kick off the DNS validation of a new hostname.

As I mentioned I can replicate this with ANY host name including the ones currently working because the certs were provisioned with HTTP validation.

The issue has to be with the DNS validation steps, not the actual proxy config. I’m just not sure what I’m missing.

Bruce5051 · May 7, 2024, 5:20pm

@denverpedro sorry but I’m getting anywhere in this.

Kindly wait for more knowledgeable Caddy community volunteers to assist.

francislavoie · May 7, 2024, 5:22pm

That’s just cause you’re clicking on links that expect to receive actual data in the request. Clicking on links does GET requests, but those expect POST etc. They’re not meant to be hit by users, only by your ACME client (Caddy). Don’t worry about drilling down those links (unless you’re looking to write an ACME client), you’ll hit dead ends.

This sounds like an API auth error to me (invalid token). Make sure the token you used in your Caddyfile is correct. Double check that you didn’t add any extra syntax or special characters to it.

denverpedro · May 7, 2024, 5:52pm

Thank you, yes I’ve validated the token several times. I’ve even re-created new tokens. No luck unfortunately.

francislavoie · May 7, 2024, 6:40pm

Are you using env vars to pass it to Caddy, or are you writing it as-is in your Caddyfile?

denverpedro · May 7, 2024, 6:52pm

Writing it in the Caddyfile.

francislavoie · May 8, 2024, 2:55am

I don’t use that DNS provider, so there’s not much else I can suggest to try.

You’ll probably need to reach out to the plugin author via GitHub and/or Linode’s technical support.

denverpedro · May 8, 2024, 5:27pm

Well… I actually work for Akamai (the parent company of Linode) so I don’t think it’s a Linode issue from my troubleshooting also why I can’t switch to a competitor (CloudFlare) for my DNS. I am now actively looking at Traefik because the DNS API integration works flawlessly for that, I just liked the easy Caddy configuration as opposed to the more complicated Traefik setup. But I appreciate everyone’s help!

francislavoie · May 8, 2024, 7:48pm

Then maybe look into how the plugin is implemented. There might be a bug there.

system · June 7, 2024, 7:48pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.