Caddy `on_demand_tls` with Cloudflare DNS

1. The problem I’m having:

My apps allows users to setup custom domains with a CNAME record. I use cloudflare DNS along with the proxy enabled and with Full encryption mode

My Caddyfile looks like the following.

{
  on_demand_tls {
    ask http://localhost:8001/caddy/ask
  }
}

https:// {
  tls {
    on_demand
  }
  reverse_proxy  localhost:8001
}

api.getnoto.app {
  reverse_proxy  127.0.0.1:8001
}

*.getnoto.page {
  reverse_proxy  127.0.0.1:8001
}

The problem is that the requests won’t work, the browser would yield ERR_SSL_VERSION_OR_CIPHER_MISMATCH error. But it works perfectly fine when I disable the DNS proxy (Which I cannot, since it exposes my real IP)

How do I make it work?

2. Error messages and/or full log output:

I couldn’t find any meaning full logs even after running Caddy in debug mode. But the following are my application logs

INFO:     172.18.0.1:42728 - "GET /caddy/ask?domain=172.31.24.112 HTTP/1.1" 403 Forbidden
INFO:     172.18.0.1:54676 - "GET /caddy/ask?domain=dummy.acme.corp HTTP/1.1" 200 OK

The first request is with CF proxy enabled, the second one is with it disabled. I am assuming caddy is not getting the correct incoming domain but it’s getting the Cloudflare IP, is there a way to configure caddy to handle this behaviour?

3. Caddy version:

v2.7.5 h1:HoysvZkLcN2xJExEepaFHK92Qgs7xAiCFydN5x5Hs6Q=

4. How I installed and ran Caddy:

It’s the default installing with no plugins, with the steps outlined here Install — Caddy Documentation

a. System environment:

Ubuntu 22.04 running on a chunky EC2 instance

b. Command:

caddy start

c. Service/unit/compose file:

d. My complete Caddy config:

{
  on_demand_tls {
    ask http://localhost:8001/caddy/ask
  }
}

https:// {
  tls {
    on_demand
  }
  handle /caddy/ask {
    respond "This path is blocked." 403
  }
  handle /healthz {
    respond "This path is blocked." 403
  }
  reverse_proxy  localhost:8001
}

api.getnoto.app {
  handle /caddy/ask {
    respond "This path is blocked." 403
  }
  handle /healthz {
    respond "This path is blocked." 403
  }
  reverse_proxy  127.0.0.1:8001
}

*.getnoto.page {
  reverse_proxy  127.0.0.1:8001
}

5. Links to relevant resources:

It’s impossible to use On-Demand TLS with an HTTP-layer proxy.

If Cloudflare had a TCP proxying mode, it would be possible to use their proxy. But they don’t do that.

So you have to turn off Cloudflare proxying.

The reason is that Cloudflare is terminating TLS when acting as a proxy, so Caddy doesn’t receive the original TLS handshake, which is what triggers cert issuance.

If Cloudflare doesn’t have a certificate for the given domain (and it wouldn’t, for all the CNAMEs) then it can’t complete the initial TLS handshake, so nothing makes it through to Caddy.

Why would this be a problem? That’s how you have to do it.

1 Like

Thanks a lot for such a quick response, as a fellow OSS maintainer, I really appreciate it.

It’s impossible to use On-Demand TLS with an HTTP-layer proxy.

Yeah, I had the same doubt, just wanted to know for sure.

Why would this be a problem? That’s how you have to do it.

It potentially exposes my app to DDOS, without the proxy there is no DDOS protection. Perhaps you’re right, there’s no alternative,. I saw there’s a SSL for SaaS offering from Cloudflare, but it’s safe to assume that is a lot of $$$$.

Regardless, thanks for the info.

You can mitigate the risk by scaling out Caddy to multiple servers. As long as they all share the same storage (e.g. Redis or whatever other storage plugin) then the Caddy instances will coordinate when issuing certs.

You should have somekind of firewall in front of Caddy so that you can react to abusive clients and block their IP for some time.

1 Like