How would I setup Caddy with Cloudflare to hide my IP using Wildcard DNS records?

zastrix · September 3, 2023, 12:22pm

1. The problem I’m having:

Hello!

I’ve moved my DNS records from Namecheap to Cloudflare and I’ve been successfully using it. The only issue I have is that when I run the dig CLI tool I can see my public IP.

I saw that Cloudflare has the check option on the DNS records to enable their proxy:

And this doesn’t work out of the box per se. If I disable it to activate DNS only mode and then get the Let’s Encrypt certificate and enable the proxy after the fact it works. But I’d like to be able to use a wildcard so I don’t need to add a new record for a new application each time. And it’s a big security issue if I disable cloudflare’s proxy to just get a certificate for a new application and then re-run it.

I did try to use the Cloudflare API for this but it didn’t seem to work with the proxy enabled (I have this setup for my LAN SSL).

2. Error messages and/or full log output:

root@Caddy:/etc/caddy# caddy run
2023/09/03 12:17:18.850 INFO    using adjacent Caddyfile
2023/09/03 12:17:18.856 WARN    Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies    {"adapter": "caddyfile", "file": "Caddyfile", "line": 3}
2023/09/03 12:17:18.865 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/09/03 12:17:18.866 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/09/03 12:17:18.866 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2023/09/03 12:17:18.867 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2023/09/03 12:17:18.868 INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2023/09/03 12:17:18.868 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/09/03 12:17:18.868 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/09/03 12:17:18.868 INFO    http    enabling automatic TLS certificate management   {"domains": ["local.armor.quest", "nextcloud.rare.armor.quest"]}
2023/09/03 12:17:18.872 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2023/09/03 12:17:18.872 INFO    serving initial configuration
2023/09/03 12:17:18.875 INFO    tls.obtain      acquiring lock  {"identifier": "local.armor.quest"}
2023/09/03 12:17:18.880 INFO    tls.obtain      lock acquired   {"identifier": "local.armor.quest"}
2023/09/03 12:17:18.881 INFO    tls.obtain      obtaining certificate   {"identifier": "local.armor.quest"}
2023/09/03 12:17:18.886 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["local.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:18.886 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["local.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:18.887 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0003f3500"}
2023/09/03 12:17:18.887 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2023/09/03 12:17:18.891 INFO    [INFO] Certificate certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.crt expired 1498h7m7.891223186s ago; cleaning up
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.crt because resource expired
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.key because resource expired
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.json because resource expired
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest because key is empty
2023/09/03 12:17:18.893 INFO    [INFO] Certificate certificates/local/proxmox.home.local/proxmox.home.local.crt expired 3645h12m45.893438333s ago; cleaning up
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local/proxmox.home.local.crt because resource expired
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local/proxmox.home.local.key because resource expired
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local/proxmox.home.local.json because resource expired
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local because key is empty
2023/09/03 12:17:18.894 INFO    tls     finished cleaning storage units
2023/09/03 12:17:18.895 INFO    tls.obtain      acquiring lock  {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.903 INFO    tls.obtain      lock acquired   {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.904 INFO    tls.obtain      obtaining certificate   {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.906 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["nextcloud.rare.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:18.907 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["nextcloud.rare.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:19.797 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "local.armor.quest", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: local.armor.quest, retry after 2023-09-04T19:44:42Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/"}
2023/09/03 12:17:19.802 INFO    tls.issuance.zerossl    waiting on internal rate limiter        {"identifiers": ["local.armor.quest"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/09/03 12:17:19.802 INFO    tls.issuance.zerossl    done waiting on internal rate limiter   {"identifiers": ["local.armor.quest"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/09/03 12:17:20.127 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "nextcloud.rare.armor.quest", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/09/03 12:17:20.632 INFO    tls.issuance.zerossl.acme_client        trying to solve challenge       {"identifier": "local.armor.quest", "challenge_type": "dns-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/09/03 12:17:28.751 INFO    tls.issuance.acme.acme_client   authorization finalized {"identifier": "nextcloud.rare.armor.quest", "authz_status": "valid"}
2023/09/03 12:17:28.751 INFO    tls.issuance.acme.acme_client   validations succeeded; finalizing order {"order": "https://acme-v02.api.letsencrypt.org/acme/order/1044260277/205911862626"}
2023/09/03 12:17:30.185 INFO    tls.issuance.acme.acme_client   successfully downloaded available certificate chains    {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/04858b0907f18b62f1e57bd7d658a74ad33d"}
2023/09/03 12:17:30.186 INFO    tls.obtain      certificate obtained successfully       {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:30.186 INFO    tls.obtain      releasing lock  {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:51.044 INFO    tls.issuance.zerossl.acme_client        authorization finalized {"identifier": "local.armor.quest", "authz_status": "valid"}
2023/09/03 12:17:51.044 INFO    tls.issuance.zerossl.acme_client        validations succeeded; finalizing order {"order": "https://acme.zerossl.com/v2/DV90/order/YAe_TcMZ0bVCMSdp8MebqQ"}
2023/09/03 12:18:07.688 INFO    tls.issuance.zerossl.acme_client        successfully downloaded available certificate chains    {"count": 1, "first_url": "https://acme.zerossl.com/v2/DV90/cert/-EgAh1jTAMAf-wZDmeqrWw"}
2023/09/03 12:18:07.689 INFO    tls.obtain      certificate obtained successfully       {"identifier": "local.armor.quest"}
2023/09/03 12:18:07.690 INFO    tls.obtain      releasing lock  {"identifier": "local.armor.quest"}

Which means that I can get the certificate for nextcloud.rare.armor.quest.

But when I try to access the page:

Which seems to be some kind of issue with routing but I’m not really sure how to fix this. When I disable the proxy option, it works just fine.

(The local.armor.quest is of not importance here as it’s just there to show that the cloudflare API works and how I set it on LAN. I can open the page just fine on my local browser (I deleted the certificate files for tests)).

3. Caddy version:

root@Caddy:/etc/caddy# caddy version
v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

a. System environment:

Proxmox with a Debian 11 installation.

b. Command:

caddy run

c. Service/unit/compose file:

The default caddy systemd file. Just ran systemctl enable caddy --now

d. My complete Caddy config:

With Let’s Encrypt and Proxy AFTER the certificate is obtained:

nextcloud.rare.armor.quest {
        encode gzip
        reverse_proxy 192.168.0.149:8080
}

# Local service
local.armor.quest {
        encode gzip
        tls {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        reverse_proxy 192.168.0.145:8080
}

My try with the proxy:

nextcloud.rare.armor.quest {
        encode gzip
        tls {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        reverse_proxy 192.168.0.149:8080
}

local.armor.quest {
        encode gzip
        tls {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        reverse_proxy 192.168.0.145:8080
}

5. Links to relevant resources:

How to use Caddy with Cloudflare's SSL settings

francislavoie · September 5, 2023, 2:45am

You need to build Caddy with the Cloudflare DNS plugin. See How to use DNS provider modules in Caddy 2

zastrix · September 5, 2023, 7:54am

Whoops, looks like I accidentally managed to miss that information in the opening thread. I did install caddy with the cloudflare DNS plugin. If you see the local.armor.quest entry in the Caddyfile it’s using the cloudflare api in both situations and it works. I just can’t seem to manage to make my public server work via the Cloudflare proxy, unless I pregenerate the certificates.

Whitestrake · September 6, 2023, 11:35pm

zastrix:

2023/09/03 12:17:18.895 INFO    tls.obtain      acquiring lock  {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.903 INFO    tls.obtain      lock acquired   {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.904 INFO    tls.obtain      obtaining certificate   {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.906 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["nextcloud.rare.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:18.907 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["nextcloud.rare.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:19.797 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "local.armor.quest", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: local.armor.quest, retry after 2023-09-04T19:44:42Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/"}
2023/09/03 12:17:19.802 INFO    tls.issuance.zerossl    waiting on internal rate limiter        {"identifiers": ["local.armor.quest"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/09/03 12:17:19.802 INFO    tls.issuance.zerossl    done waiting on internal rate limiter   {"identifiers": ["local.armor.quest"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/09/03 12:17:20.127 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "nextcloud.rare.armor.quest", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/09/03 12:17:20.632 INFO    tls.issuance.zerossl.acme_client        trying to solve challenge       {"identifier": "local.armor.quest", "challenge_type": "dns-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/09/03 12:17:28.751 INFO    tls.issuance.acme.acme_client   authorization finalized {"identifier": "nextcloud.rare.armor.quest", "authz_status": "valid"}
2023/09/03 12:17:28.751 INFO    tls.issuance.acme.acme_client   validations succeeded; finalizing order {"order": "https://acme-v02.api.letsencrypt.org/acme/order/1044260277/205911862626"}
2023/09/03 12:17:30.185 INFO    tls.issuance.acme.acme_client   successfully downloaded available certificate chains    {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/04858b0907f18b62f1e57bd7d658a74ad33d"}
2023/09/03 12:17:30.186 INFO    tls.obtain      certificate obtained successfully       {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:30.186 INFO    tls.obtain      releasing lock  {"identifier": "nextcloud.rare.armor.quest"}

Looking closer at these logs, it seems like, at least for this subdomain, DNS validation worked just fine.

Are you able to provide us logs of a fresh subdomain where you’re running into this issue?

When using DNS validation, the challenge is completed entirely out-of-band of HTTP access to the server, so Cloudflare’s proxy status should be entirely irrelevant, and the fact that putting your wildcard behind the orange cloud breaks DNS validation strongly implies that it’s not, in fact, using DNS validation at all, or something else particularly weird and unexpected is going on that I can’t really guess at and some failure logs might help us narrow down.

It is very possible to get HTTP-01 validation working behind the orange cloud, but Cloudflare has a pretty decent number of pitfalls in site configuration that can all break this, so it has to be configured fairly specifically not to interfere.

P.S. while testing you can use the staging ACME server with the global option in your Caddyfile to avoid rate limit issues while you run through troubleshooting attempts:

{
  acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

zastrix · September 7, 2023, 3:30pm

Hm. Yeah that’s what I thought regarding the DNS challenge. It does seem to be completed correctly but it’s probably something to do with cloudflare and my specific setup. My homelab is currently with me (as I moved my living location) and in a week or so should be in my new apartment with a static IP address. Currently I’m using an SSH tunnel I’ve made to bypass the fact that I only have a dynamic IP so that might be the cause of this whole issue.

I’m not going to look too much into this while running the SSH tunnel but here’s the service file for the tunnel if that peaked your interest (again, it’s only a dirty solution because I really needed the servers which I self host on a thinclient):

# service file @ /etc/systemd/system/oci-tunnel@.service 
[Unit]
Description=SSH tunnel on OCI for port %i
After=network.target

[Service]
ExecStart=/usr/bin/ssh -NT -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -R %i:0.0.0.0:%i root@server-ip

# Restart every >2 seconds to avoid StartLimitInterval failure
RestartSec=5
Restart=always

[Install]
WantedBy=multi-user.target

root@Caddy:~# systemctl enable --now oci-tunnel@443
root@Caddy:~# systemctl enable --now oci-tunnel@80

zastrix · October 1, 2023, 7:27pm

Ok, so after seeing some stuff, I saw a few things which I needed to address.

Firstly I had to change my SSL/TLS mode to Full (strict):

SSL/TLS option in Cloudflare Panel

This allowed me to run proxied connection with a single subdomain: heimdall.armor.quest:

(cloudflare) {
        tls {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
}

heimdall.armor.quest {
        encode gzip
        import cloudflare
        reverse_proxy 192.168.0.145:8080
}

This did not help with my usecase as I wanted to run really big subdomains.

After digging around I found that I need to essentially use Total TLS from cloudflare and for that I need to buy Advanced Certificate Manager. Which turns out to be around $10 / month.

If I were to upload my custom certificated it would be $200 a month (wowza).

I’m planning to get it in 2 weeks as it’s honestly worth it that I don’t want to mess around with getting DDoS’d. I really honestly just like my subdomains and I know exactly what is on which server so I’m not just going to use one-level subdomain.

I’d rather not close the thread yet as I’d like to go through with this and maybe help with future people who get to the same problem as me

balloon · October 2, 2023, 5:45am

I just saw this post.

At this stage there is already a problem. Certificates issued by Cloudflare are limited to example.com and *.example.com wildcard certificates.
You’re proxying *.rate.example.com , but Cloudflare won’t issue this certificate. (That’s the spec for wildcard certificates) This is the root cause of the error.

This is a Cloudflare level issue. It’s not a Caddy level issue.

Whitestrake · October 3, 2023, 12:11am

Ahh, yeah! @balloon, I needed to see your comment before I really understood what @zastrix was getting at: Cloudflare deciding not to handle TLS for sub-subdomains unless you pay for their product somehow. So the entire time Caddy’s been absolutely fine - got the cert just fine via DNS validation, was serving requests just fine. It was just Cloudflare at the edge that caused this issue.

To be brutally honest, from a cost standpoint, it actually makes more sense to buy a few entirely new domains to spare than it does to pay Cloudflare even $10/mo; Cloudflare’s own registrar service sells .com domains at less than US$10/yr, so unless you need more than 12 of them, you’d pay less than Advanced Certificate Manager.

The value proposition of entirely new domains goes way up, too, if you’re weighing the $200/mo… Yikes. That’s not a reasonable fee just to deal with this specific issue unless you’re getting some serious other value out of it.

zastrix · October 3, 2023, 8:40am

It would make sense from a financial point of view to just straight-out buy domains and use them, that’s definitely true.

But I really just use one domain and mess around with it for my homelab and VPSs which I got for free.

It’s mostly a personal thing as I like messing around with the names. I.E. my Vaultwarden is hosted at: https://vault.of.the.rare.armor.quest where rare is a codename for my VPS (it’s easier mentally to know what is where). Not to mention it’s already used on all of my devices and my family uses it as well.

In the future when I decide to change up my services to use different domains I would opt-in for just having… well… multiple domains.

francislavoie · October 3, 2023, 8:43am

You could just use dashes instead of dots for your domains past the 2nd level, e.g. vault-of-the-rare.armor.quest or something like that. That way, the wildcard cert still applies.

balloon · October 3, 2023, 9:14am

Yes. Cloudflare cannot issue a certificate for vault.of.the.rare.armor.quest. It should be vault-of-the-rare.armor.quest or vaultoftherare.armor.quest instead as suggested by francislavoie.
If you proxy with Cloudflare, it’s tricky. (If you set it to DNS only, this problem will not occur)

system · November 2, 2023, 9:14am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.