How would I setup Caddy with Cloudflare to hide my IP using Wildcard DNS records?

Help
1

1. The problem I’m having:

Hello!

I’ve moved my DNS records from Namecheap to Cloudflare and I’ve been successfully using it. The only issue I have is that when I run the dig CLI tool I can see my public IP.

I saw that Cloudflare has the check option on the DNS records to enable their proxy:

image
image1289×96 6.78 KB

And this doesn’t work out of the box per se. If I disable it to activate DNS only mode and then get the Let’s Encrypt certificate and enable the proxy after the fact it works. But I’d like to be able to use a wildcard so I don’t need to add a new record for a new application each time. And it’s a big security issue if I disable cloudflare’s proxy to just get a certificate for a new application and then re-run it.

I did try to use the Cloudflare API for this but it didn’t seem to work with the proxy enabled (I have this setup for my LAN SSL).

2. Error messages and/or full log output:

root@Caddy:/etc/caddy# caddy run
2023/09/03 12:17:18.850 INFO    using adjacent Caddyfile
2023/09/03 12:17:18.856 WARN    Caddyfile input is not formatted; run 'caddy fmt --overwrite' to fix inconsistencies    {"adapter": "caddyfile", "file": "Caddyfile", "line": 3}
2023/09/03 12:17:18.865 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["//localhost:2019", "//[::1]:2019", "//127.0.0.1:2019"]}
2023/09/03 12:17:18.866 INFO    http.auto_https server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2023/09/03 12:17:18.866 INFO    http.auto_https enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2023/09/03 12:17:18.867 INFO    http    enabling HTTP/3 listener        {"addr": ":443"}
2023/09/03 12:17:18.868 INFO    failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/quic-go/quic-go/wiki/UDP-Buffer-Sizes for details.
2023/09/03 12:17:18.868 INFO    http.log        server running  {"name": "srv0", "protocols": ["h1", "h2", "h3"]}
2023/09/03 12:17:18.868 INFO    http.log        server running  {"name": "remaining_auto_https_redirects", "protocols": ["h1", "h2", "h3"]}
2023/09/03 12:17:18.868 INFO    http    enabling automatic TLS certificate management   {"domains": ["local.armor.quest", "nextcloud.rare.armor.quest"]}
2023/09/03 12:17:18.872 INFO    autosaved config (load with --resume flag)      {"file": "/root/.config/caddy/autosave.json"}
2023/09/03 12:17:18.872 INFO    serving initial configuration
2023/09/03 12:17:18.875 INFO    tls.obtain      acquiring lock  {"identifier": "local.armor.quest"}
2023/09/03 12:17:18.880 INFO    tls.obtain      lock acquired   {"identifier": "local.armor.quest"}
2023/09/03 12:17:18.881 INFO    tls.obtain      obtaining certificate   {"identifier": "local.armor.quest"}
2023/09/03 12:17:18.886 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["local.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:18.886 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["local.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:18.887 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc0003f3500"}
2023/09/03 12:17:18.887 INFO    tls     cleaning storage unit   {"description": "FileStorage:/root/.local/share/caddy"}
2023/09/03 12:17:18.891 INFO    [INFO] Certificate certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.crt expired 1498h7m7.891223186s ago; cleaning up
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.crt because resource expired
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.key because resource expired
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest/proxmox.rare.armor.quest.json because resource expired
2023/09/03 12:17:18.891 INFO    [INFO] Deleting certificates/acme-v02.api.letsencrypt.org-directory/proxmox.rare.armor.quest because key is empty
2023/09/03 12:17:18.893 INFO    [INFO] Certificate certificates/local/proxmox.home.local/proxmox.home.local.crt expired 3645h12m45.893438333s ago; cleaning up
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local/proxmox.home.local.crt because resource expired
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local/proxmox.home.local.key because resource expired
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local/proxmox.home.local.json because resource expired
2023/09/03 12:17:18.893 INFO    [INFO] Deleting certificates/local/proxmox.home.local because key is empty
2023/09/03 12:17:18.894 INFO    tls     finished cleaning storage units
2023/09/03 12:17:18.895 INFO    tls.obtain      acquiring lock  {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.903 INFO    tls.obtain      lock acquired   {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.904 INFO    tls.obtain      obtaining certificate   {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:18.906 INFO    tls.issuance.acme       waiting on internal rate limiter        {"identifiers": ["nextcloud.rare.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:18.907 INFO    tls.issuance.acme       done waiting on internal rate limiter   {"identifiers": ["nextcloud.rare.armor.quest"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": ""}
2023/09/03 12:17:19.797 ERROR   tls.obtain      could not get certificate from issuer   {"identifier": "local.armor.quest", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "HTTP 429 urn:ietf:params:acme:error:rateLimited - Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: local.armor.quest, retry after 2023-09-04T19:44:42Z: see https://letsencrypt.org/docs/duplicate-certificate-limit/"}
2023/09/03 12:17:19.802 INFO    tls.issuance.zerossl    waiting on internal rate limiter        {"identifiers": ["local.armor.quest"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/09/03 12:17:19.802 INFO    tls.issuance.zerossl    done waiting on internal rate limiter   {"identifiers": ["local.armor.quest"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "caddy@zerossl.com"}
2023/09/03 12:17:20.127 INFO    tls.issuance.acme.acme_client   trying to solve challenge       {"identifier": "nextcloud.rare.armor.quest", "challenge_type": "dns-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2023/09/03 12:17:20.632 INFO    tls.issuance.zerossl.acme_client        trying to solve challenge       {"identifier": "local.armor.quest", "challenge_type": "dns-01", "ca": "https://acme.zerossl.com/v2/DV90"}
2023/09/03 12:17:28.751 INFO    tls.issuance.acme.acme_client   authorization finalized {"identifier": "nextcloud.rare.armor.quest", "authz_status": "valid"}
2023/09/03 12:17:28.751 INFO    tls.issuance.acme.acme_client   validations succeeded; finalizing order {"order": "https://acme-v02.api.letsencrypt.org/acme/order/1044260277/205911862626"}
2023/09/03 12:17:30.185 INFO    tls.issuance.acme.acme_client   successfully downloaded available certificate chains    {"count": 2, "first_url": "https://acme-v02.api.letsencrypt.org/acme/cert/04858b0907f18b62f1e57bd7d658a74ad33d"}
2023/09/03 12:17:30.186 INFO    tls.obtain      certificate obtained successfully       {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:30.186 INFO    tls.obtain      releasing lock  {"identifier": "nextcloud.rare.armor.quest"}
2023/09/03 12:17:51.044 INFO    tls.issuance.zerossl.acme_client        authorization finalized {"identifier": "local.armor.quest", "authz_status": "valid"}
2023/09/03 12:17:51.044 INFO    tls.issuance.zerossl.acme_client        validations succeeded; finalizing order {"order": "https://acme.zerossl.com/v2/DV90/order/YAe_TcMZ0bVCMSdp8MebqQ"}
2023/09/03 12:18:07.688 INFO    tls.issuance.zerossl.acme_client        successfully downloaded available certificate chains    {"count": 1, "first_url": "https://acme.zerossl.com/v2/DV90/cert/-EgAh1jTAMAf-wZDmeqrWw"}
2023/09/03 12:18:07.689 INFO    tls.obtain      certificate obtained successfully       {"identifier": "local.armor.quest"}
2023/09/03 12:18:07.690 INFO    tls.obtain      releasing lock  {"identifier": "local.armor.quest"}

Which means that I can get the certificate for nextcloud.rare.armor.quest.

But when I try to access the page:

image
image1213×648 51.4 KB

Which seems to be some kind of issue with routing but I’m not really sure how to fix this. When I disable the proxy option, it works just fine.

(The local.armor.quest is of not importance here as it’s just there to show that the cloudflare API works and how I set it on LAN. I can open the page just fine on my local browser (I deleted the certificate files for tests)).

3. Caddy version:

root@Caddy:/etc/caddy# caddy version
v2.7.4 h1:J8nisjdOxnYHXlorUKXY75Gr6iBfudfoGhrJ8t7/flI=

4. How I installed and ran Caddy:

a. System environment:

Proxmox with a Debian 11 installation.

b. Command:

caddy run

c. Service/unit/compose file:

The default caddy systemd file. Just ran systemctl enable caddy --now

d. My complete Caddy config:

With Let’s Encrypt and Proxy AFTER the certificate is obtained:

nextcloud.rare.armor.quest {
        encode gzip
        reverse_proxy 192.168.0.149:8080
}

# Local service
local.armor.quest {
        encode gzip
        tls {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        reverse_proxy 192.168.0.145:8080
}

My try with the proxy:

nextcloud.rare.armor.quest {
        encode gzip
        tls {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        reverse_proxy 192.168.0.149:8080
}

local.armor.quest {
        encode gzip
        tls {
                dns cloudflare {env.CLOUDFLARE_API_KEY}
        }
        reverse_proxy 192.168.0.145:8080
}

5. Links to relevant resources:

2

You need to build Caddy with the Cloudflare DNS plugin. See How to use DNS provider modules in Caddy 2

1 Like
3

Whoops, looks like I accidentally managed to miss that information in the opening thread. I did install caddy with the cloudflare DNS plugin. If you see the local.armor.quest entry in the Caddyfile it’s using the cloudflare api in both situations and it works. I just can’t seem to manage to make my public server work via the Cloudflare proxy, unless I pregenerate the certificates.

4

Looking closer at these logs, it seems like, at least for this subdomain, DNS validation worked just fine.

Are you able to provide us logs of a fresh subdomain where you’re running into this issue?

When using DNS validation, the challenge is completed entirely out-of-band of HTTP access to the server, so Cloudflare’s proxy status should be entirely irrelevant, and the fact that putting your wildcard behind the orange cloud breaks DNS validation strongly implies that it’s not, in fact, using DNS validation at all, or something else particularly weird and unexpected is going on that I can’t really guess at and some failure logs might help us narrow down.

It is very possible to get HTTP-01 validation working behind the orange cloud, but Cloudflare has a pretty decent number of pitfalls in site configuration that can all break this, so it has to be configured fairly specifically not to interfere.

P.S. while testing you can use the staging ACME server with the global option in your Caddyfile to avoid rate limit issues while you run through troubleshooting attempts:

{
  acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}
5

Hm. Yeah that’s what I thought regarding the DNS challenge. It does seem to be completed correctly but it’s probably something to do with cloudflare and my specific setup. My homelab is currently with me (as I moved my living location) and in a week or so should be in my new apartment with a static IP address. Currently I’m using an SSH tunnel I’ve made to bypass the fact that I only have a dynamic IP so that might be the cause of this whole issue.

I’m not going to look too much into this while running the SSH tunnel but here’s the service file for the tunnel if that peaked your interest (again, it’s only a dirty solution because I really needed the servers which I self host on a thinclient):

# service file @ /etc/systemd/system/oci-tunnel@.service 
[Unit]
Description=SSH tunnel on OCI for port %i
After=network.target

[Service]
ExecStart=/usr/bin/ssh -NT -o ServerAliveInterval=60 -o ExitOnForwardFailure=yes -R %i:0.0.0.0:%i root@server-ip

# Restart every >2 seconds to avoid StartLimitInterval failure
RestartSec=5
Restart=always

[Install]
WantedBy=multi-user.target

root@Caddy:~# systemctl enable --now oci-tunnel@443
root@Caddy:~# systemctl enable --now oci-tunnel@80
1 Like