Caddy & Tailscale & Cloudflare, Oh my! (certificate issue)

1. The problem I’m having:

I was able to get Caddy working as a reverse proxy for a single local service with Tailscale, but I have multiple service running locally. The video from Tailscale made it look so easy, but I haven’t been able duplicate their success using subdomains to reverse proxy to different services. After spending hours going through support forums in search of answers, I figured it’s finally time to get some direct help on my situation.

Here’s the steps I’ve taken:

  • I have a test domain from Namecheap and used Cloudflare to setup a CNAME (tried both *.app and test for names)
  • Created API token with Zone:Read and DNS:Edit
  • I added TS_PERMIT_CERT_UID=caddy to /etc/default/tailscaled
  • HTTPS and MagicDNS are both turned on in Tailscale and it shows the cert for the machine is valid for three months
  • Currently using the cloudflare version (module: dns.providers.cloudflare)

If I go to the Tailscale domain for the machine and include the port it works but it doesn’t work if I try and use test.app.lunaticmallard.com.

As you can probably tell, I’m still very new to this so please let me know if there is any additional information I can provide to help diagnose the issue.

2. Error messages and/or full log output:

Aug 31 18:12:24 lin caddy[147677]: {"level":"debug","ts":1725142344.3353295,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"100.87.73.93","remote_port":"59336","server_name":"lin.tail2cd30c.ts.net","remote":"100.87.73.93:59336","identifier":"lin.tail2cd30c.ts.net","cipher_suites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
Aug 31 18:12:24 lin caddy[147677]: {"level":"debug","ts":1725142344.335465,"logger":"http.stdlib","msg":"http: TLS handshake error from 100.87.73.93:59336: no certificate available for 'lin.tail2cd30c.ts.net'"}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.3394558,"logger":"events","msg":"event","name":"tls_get_certificate","id":"0c18a51f-019a-4c5d-91da-529b3eb1b67a","origin":"tls","data":{"client_hello":{"CipherSuites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"ServerName":"lin.tail2cd30c.ts.net","SupportedCurves":[29,23,24,25,256,257],"SupportedPoints":"AA==","SignatureSchemes":[1027,1283,1539,2052,2053,2054,1025,1281,1537,515,513],"SupportedProtos":["h2","http/1.1"],"SupportedVersions":[772,771],"RemoteAddr":{"IP":"100.87.73.93","Port":59350,"Zone":""},"LocalAddr":{"IP":"100.87.73.93","Port":443,"Zone":""}}}}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.339474,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"lin.tail2cd30c.ts.net"}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.3394787,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.tail2cd30c.ts.net"}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.3394816,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.ts.net"}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.3394847,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*.net"}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.3394876,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*.*.*"}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.3394926,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"100.87.73.93","remote_port":"59350","server_name":"lin.tail2cd30c.ts.net","remote":"100.87.73.93:59350","identifier":"lin.tail2cd30c.ts.net","cipher_suites":[4865,4867,4866,49195,49199,52393,52392,49196,49200,49162,49161,49171,49172,156,157,47,53],"cert_cache_fill":0,"load_or_obtain_if_necessary":true,"on_demand":false}
Aug 31 18:12:27 lin caddy[147677]: {"level":"debug","ts":1725142347.3395238,"logger":"http.stdlib","msg":"http: TLS handshake error from 100.87.73.93:59350: no certificate available for 'lin.tail2cd30c.ts.net'"}

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

Honestly, I can’t remember but I’m pretty sure it was xcaddy because I needed to install Go so I could get the cloudflare module.

a. System environment:

Linux 24.04. As a test I’ve been running a docker container with nginx, which is linked to port 8080. I have also tried RStudio server which is run locally with the same results.

b. Command:

sudo systemctl <verb> caddy

c. Service/unit/compose file:

To create the test nginx container

docker run -it --rm -d -p 8080:80 --name web nginx

d. My complete Caddy config:

{
        debug
}

cloudflare {
        tls {
                dns cloudflare <api token>
        }
}
#nginx test
test.app.lunaticmallard.com {
        reverse_proxy localhost:8080
}

5. Links to relevant resources:

Howdy @jonbry, welcome to the Caddy community!

The goal of the video is to use public DNS to serve a CNAME to your ts.net address.

I queried the domains test.app.lunaticmallard.com, app.lunaticmallard.com, and lunaticmallard.com, and didn’t see any DNS records for these.

The purpose as illustrated in the video is to make it so that a request from a computer outside your tailnet will see a CNAME pointing to lin.tail2cd30c.ts.net but won’t be able to resolve any further to an actual IP address. Meanwhile, a computer with access to your tailnet will be able to follow this chain to lin.tail2cd30c.ts.net’s actual tailnet IP (using Tailscale’s internal DNS) and make the request directly to Caddy via your own domain name.

So, unless you intend to configure split horizon DNS, your next step is to make sure your lunaticmallard.com subdomains have proper CNAMEs configured in public DNS.

1 Like

Hi @Whitestrake!

Thanks for getting back to me on my issue with getting Tailscale working with Caddy. I apologize as I forgot to update my post since continuing to troubleshoot the issue earlier today. I saw some people had success using an A record and others with a CNAME for each subdomain, so I tried to both options this morning. Right now, I believe I have a CNAME record nginx.lunaticmallard.com on Cloudflare that is pointing to the qualified Tailscale domain, but still no luck.

Does everything else look ok?

;; ANSWER SECTION:
nginx.lunaticmallard.com. 60	IN	CNAME	lin.tail2cd30c.ts.net.

Looks good to me.

What’s your current Caddyfile?

What happens when you browse to nginx.lunaticmallard.com?

1 Like

Here’s my current Caddyfile

{
        debug
}

(cloudflare) {
        tls {
                dns cloudflare <api_token>
        }
}

nginx.lunaticmallard.com {
        reverse_proxy localhost:8080
}

When I go to nginx.lunaticmallard.com, I get “Secure Connection Failed…Error code: SSL_ERROR_INTERNAL_ERROR_ALERT”.

Here’s the current log errors:

Sep 01 20:36:37 lin caddy[41388]: {"level":"error","ts":1725237397.0319474,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"nginx.lunaticmallard.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for nginx.lunaticmallard.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for nginx.lunaticmallard.com - check that a DNS record exists for this domain"}
Sep 01 20:36:37 lin caddy[41388]: {"level":"debug","ts":1725237397.031978,"logger":"events","msg":"event","name":"cert_failed","id":"27282a26-b4c7-4f02-9055-0904caa97702","origin":"tls","data":{"error":{},"identifier":"nginx.lunaticmallard.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
Sep 01 20:36:37 lin caddy[41388]: {"level":"error","ts":1725237397.0319862,"logger":"tls.obtain","msg":"will retry","error":"[nginx.lunaticmallard.com] Obtain: [nginx.lunaticmallard.com] solving challenge: nginx.lunaticmallard.com: [nginx.lunaticmallard.com] authorization failed: HTTP 400 urn:ietf:params:acme:error:dns - DNS problem: NXDOMAIN looking up A for nginx.lunaticmallard.com - check that a DNS record exists for this domain; DNS problem: NXDOMAIN looking up AAAA for nginx.lunaticmallard.com - check that a DNS record exists for this domain (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":2.824712299,"max_duration":2592000}

You’ve configured this snippet, but you haven’t included it in your site.

That means you haven’t actually configured nginx.lunaticmallard.com to acquire a cert via DNS challenge. LetsEncrypt is trying to connect to carry out a HTTP or TLS-ALPN challenge and naturally cannot proceed, because this domain is only resolveable via your tailnet and LetsEncrypt is not on your tailnet.

Include the snippet in your site with the directive import cloudflare or configure the DNS challenge in your global options instead in order to activate it properly.

1 Like

Hm. so it should look like the following:

{
        debug
}

(cloudflare) {
        tls {
                dns cloudflare <api_token>
        }
}

nginx.lunaticmallard.com {
        reverse_proxy localhost:8080
        import cloudflare
}


Here’s the logs with the updated Caddyfile after restarting Caddy:

Sep 01 20:47:57 lin caddy[41760]: {"level":"info","ts":1725238077.5651088,"logger":"tls.issuance.acme","msg":"using ACME account","account_id":"https://acme-v02.api.letsencrypt.org/acme/acct/1905078456","account_co
ntact":[]}
Sep 01 20:47:57 lin caddy[41760]: {"level":"debug","ts":1725238077.8851242,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"GET","url":"https://acme-v02.api.letsencrypt.org/directory","header
s":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["746"],"Content-Type":["application/json"],"Date":["Mon, 02 Sep
 2024 00:47:57 GMT"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 01 20:47:57 lin caddy[41760]: {"level":"debug","ts":1725238077.8852324,"logger":"tls.issuance.acme.acme_client","msg":"creating order","account":"https://acme-v02.api.letsencrypt.org/acme/acct/1905078456","iden
tifiers":["nginx.lunaticmallard.com"]}
Sep 01 20:47:57 lin caddy[41760]: {"level":"debug","ts":1725238077.969506,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"HEAD","url":"https://acme-v02.api.letsencrypt.org/acme/new-nonce","h
eaders":{"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Cache-Control":["public, max-age=0, no-cache"],"Date":["Mon, 02 Sep 2024 00:47:57 GMT"],"Link":["<https://acme-v02.api.lets
encrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["r6at1aZYb7H3X6NWcSxxnZKi8Hg9htylwCQMSmmh0lUWAHWykC8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_co
de":200}
Sep 01 20:47:58 lin caddy[41760]: {"level":"debug","ts":1725238078.1160476,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/new-order","
headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1905078456"],"Cache-Control":["public, max-age=0, no-cache"]
,"Content-Length":["350"],"Content-Type":["application/json"],"Date":["Mon, 02 Sep 2024 00:47:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Location":["https://acme-v02.api.let
sencrypt.org/acme/order/1905078456/301506614646"],"Replay-Nonce":["tCF_ahtplUSRwMh-PJhwd-1uDWxpfJrbtaorJMNxXR-kaXW-JW8"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]}
,"status_code":201}
Sep 01 20:47:58 lin caddy[41760]: {"level":"debug","ts":1725238078.2097273,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/398164546106","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1905078456"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["808"],"Content-Type":["application/json"],"Date":["Mon, 02 Sep 2024 00:47:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["r6at1aZYnyEcMgZQCZhcV_R3WvGfQcXqQCwsXlYZgCoi4xZQgqM"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 01 20:47:58 lin caddy[41760]: {"level":"debug","ts":1725238078.2098317,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"http-01"}
Sep 01 20:47:58 lin caddy[41760]: {"level":"debug","ts":1725238078.2098372,"logger":"tls.issuance.acme.acme_client","msg":"no solver configured","challenge_type":"tls-alpn-01"}
Sep 01 20:47:58 lin caddy[41760]: {"level":"info","ts":1725238078.2098413,"logger":"tls.issuance.acme.acme_client","msg":"trying to solve challenge","identifier":"nginx.lunaticmallard.com","challenge_type":"dns-01","ca":"https://acme-v02.api.letsencrypt.org/directory"}
Sep 01 20:47:58 lin caddy[41760]: {"level":"error","ts":1725238078.4655957,"logger":"tls.issuance.acme.acme_client","msg":"cleaning up solver","identifier":"nginx.lunaticmallard.com","challenge_type":"dns-01","error":"no memory of presenting a DNS record for \"_acme-challenge.nginx.lunaticmallard.com\" (usually OK if presenting also failed)"}
Sep 01 20:47:58 lin caddy[41760]: {"level":"debug","ts":1725238078.5583718,"logger":"tls.issuance.acme.acme_client","msg":"http request","method":"POST","url":"https://acme-v02.api.letsencrypt.org/acme/authz-v3/398164546106","headers":{"Content-Type":["application/jose+json"],"User-Agent":["Caddy/2.8.4 CertMagic acmez (linux; amd64)"]},"response_headers":{"Boulder-Requester":["1905078456"],"Cache-Control":["public, max-age=0, no-cache"],"Content-Length":["812"],"Content-Type":["application/json"],"Date":["Mon, 02 Sep 2024 00:47:58 GMT"],"Link":["<https://acme-v02.api.letsencrypt.org/directory>;rel=\"index\""],"Replay-Nonce":["r6at1aZYVAiXVPOB26ykxgibW30Q3N553rCVbg4zTYt7nEYBua0"],"Server":["nginx"],"Strict-Transport-Security":["max-age=604800"],"X-Frame-Options":["DENY"]},"status_code":200}
Sep 01 20:47:58 lin caddy[41760]: {"level":"error","ts":1725238078.5584497,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"nginx.lunaticmallard.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[nginx.lunaticmallard.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.nginx.lunaticmallard.com\": unexpected response code 'NOTIMP' for nginx.lunaticmallard.com. (order=https://acme-v02.api.letsencrypt.org/acme/order/1905078456/301506614646) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
Sep 01 20:47:58 lin caddy[41760]: {"level":"debug","ts":1725238078.5584955,"logger":"events","msg":"event","name":"cert_failed","id":"f5c04d2c-2c45-4ab3-8177-450cde1e863f","origin":"tls","data":{"error":{},"identifier":"nginx.lunaticmallard.com","issuers":["acme-v02.api.letsencrypt.org-directory"],"renewal":false}}
Sep 01 20:47:58 lin caddy[41760]: {"level":"error","ts":1725238078.558504,"logger":"tls.obtain","msg":"will retry","error":"[nginx.lunaticmallard.com] Obtain: [nginx.lunaticmallard.com] solving challenges: presenting for challenge: could not determine zone for domain \"_acme-challenge.nginx.lunaticmallard.com\": unexpected response code 'NOTIMP' for nginx.lunaticmallard.com. (order=https://acme-v02.api.letsencrypt.org/acme/order/1905078456/301506614646) (ca=https://acme-v02.api.letsencrypt.org/directory)","attempt":1,"retrying_in":60,"elapsed":0.993898593,"max_duration":2592000}

I pointed a CNAME from one of my own domains to one of my tailnet machines and set up a Caddy instance to complete a DNS challenge. I didn’t have any trouble with that process on my end, everything went through A-OK, I got my cert on the first try, and was able to curl the public domain and get a response from my Caddy instance.

Given that you’re getting DNS issues, my only conclusion here is that your local DNS resolution is misconfigured or broken.

You can tell Caddy to skip local DNS resolution and use a specific server with the resolvers subdirective: tls (Caddyfile directive) — Caddy Documentation

It would look something like this:

(cloudflare) {
        tls {
                dns cloudflare <api_token>
                resolvers 1.1.1.1 1.0.0.1
        }
}
3 Likes

It works! Thank you so much @Whitestrake!

One final question, can I replace my current CNAME with a wildcard (*) and then have have multiple reverse proxies to different local services? For example:

(cloudflare) {
        tls {
                dns cloudflare <api_token>
                resolvers 1.1.1.1 1.0.0.1
        }
}

#nginx
nginx.lunaticmallard.com {
        reverse_proxy localhost:8080
        import cloudflare
}

#rstudio
rstudio.lunaticmallard.com {
       reverse_proxy localhost:8787
       import cloudflare
}

Would having a “broken DNS” cause any issues with this setup?

Thank you for all your help!

EDIT: I change the CNAME to * and the subdomains in the Caddyfile worked perfectly! Thanks again for all your help!

1 Like