HTTPS behind Cloudflare proxy

1. Caddy version (caddy version):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

(Built with xcaddy + Redis.)

2. How I run Caddy:

caddy start

a. System environment:

Debian 10. 16 GB memory.

b. Command:

sudo ./caddy start

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

I’ve redacted a couple bits of semi-sensitive info with “…”.

{
	on_demand_tls {
		ask https://micro.blog/pages/ssl/check
		interval 2m
		burst 100
	}
	
	acme_ca https://acme.zerossl.com/v2/DV90
	email ...
	
	storage redis {
		host "..."
		port 9000
		password ""
		db 1
	}

	log {
		output file /var/log/caddy-main.log {
			roll_keep_for 5d
		}
		format console
	}
}

:443 {
	tls {
		on_demand
		issuer zerossl ABCDEFGHIJK {
			disable_tlsalpn_challenge
		}
	}
	
	root * /home/web/sites/{host}
	file_server

	log {
		output file /var/log/caddy-access.log {
			roll_keep_for 5d
		}
		format console
	}
	
	handle_errors {
		@502 {
			expression {http.error.status_code} == 502
		}
		handle @502 {
			rewrite * 502.html
			file_server
		}
		handle {
			# we do this to route 404s back to our web app for special handling
			rewrite * /pages/migration/redirect{path}?domain={host}
			reverse_proxy micro.blog {
				header_up Host micro.blog
			}
		}
	}
}

3. The problem I’m having:

Hi! I run Micro.blog, a blog hosting service, and we’ve been migrating from Nginx to Caddy. My approach has been to use a load balancer (on Linode) to direct all port 80 traffic and a small percentage of port 443 traffic to Caddy, to slowly ramp up the HTTPS certificates in Caddy.

For the most part, this has worked wonderfully, but there are some custom domains from our customers that aren’t successfully getting new HTTPS certificates. Most of the problem seems to be from domain names routed through Cloudflare. I’ve read all the forum posts about this and have continued to tweak my Caddyfile, but still don’t have a solution that works all the time. The domains that fail, repeatedly fail.

These errors show up in the Caddy log with “could not get certificate from issuer” and “context deadline exceeded”.

Note that we don’t have any control over Cloudflare for these domains. I thought that forcing port 80 authentication would solve this, but no luck. Thanks in advance for any suggestions!

4. Error messages and/or full log output:

I’ve redacted the domain name to “example.com” here because I don’t want to share our customers’ domain names. There’s nothing special about the domain name other than it is pointed to Cloudflare DNS.

1.629637776267173e+09	info	tls.on_demand	obtaining new certificate	{"server_name": "example.com"}
1.6296377762703438e+09	info	tls.obtain	acquiring lock	{"identifier": "example.com"}
1.6296377762708578e+09	info	tls.obtain	lock acquired	{"identifier": "example.com"}
1.629637776272935e+09	info	tls.issuance.acme	waiting on internal rate limiter	{"identifiers": ["example.com"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "admin@micro.blog"}
1.6296377762729974e+09	info	tls.issuance.acme	done waiting on internal rate limiter	{"identifiers": ["example.com"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "admin@micro.blog"}
1.6296377767547162e+09	info	tls.issuance.acme.acme_client	trying to solve challenge	{"identifier": "example.com", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}
1.6296378662696438e+09	error	tls.issuance.acme.acme_client	deactivating authorization	{"identifier": "example.com", "authz": "https://acme.zerossl.com/v2/DV90/authz/Bigx1fMATvYx9AnXWx5RYw", "error": "request to https://acme.zerossl.com/v2/DV90/authz/Bigx1fMATvYx9AnXWx5RYw failed after 1 attempts: context deadline exceeded"}
1.6296378662697086e+09	error	tls.obtain	could not get certificate from issuer	{"identifier": "example.com", "issuer": "acme.zerossl.com-v2-DV90", "error": "[example.com] solving challenges: [example.com] context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/vICNCQMlEyXQ8JSnFIUyuw) (ca=https://acme.zerossl.com/v2/DV90)"}
1.6296378662697346e+09	error	tls.obtain	will retry	{"error": "[example.com] Obtain: [example.com] solving challenges: [example.com] context deadline exceeded (order=https://acme.zerossl.com/v2/DV90/order/vICNCQMlEyXQ8JSnFIUyuw) (ca=https://acme.zerossl.com/v2/DV90)", "attempt": 1, "retrying_in": 60, "elapsed": 89.998822121, "max_duration": 2592000}
1.6296378662697573e+09	info	tls.obtain	releasing lock	{"identifier": "example.com"}

5. What I already tried:

  • Added ask in Caddyfile.
  • Changed config to only use ZeroSSL. We were getting rate-limited by Let’s Encrypt.
  • Added a ZeroSSL API key.
  • Routed all port 80 to Caddy.
  • Tried 100% of port 443 to Caddy or 5%.
  • Disabled TLS-ALPN.

6. Links to relevant resources:

n/a

Are you saying that your customer’s domains are using Cloudflare? Or are you saying that you’re using Cloudflare?

If the former, I think you’ll just need to explain to your customers to not use Cloudflare for the domains they point to your servers (or to turn off proxying, I think it’s the “grey cloud” option? I don’t use Cloudflare myself).

If Cloudflare is intercepting the requests, then they’ll be in the way of the ACME challenge requests reaching your servers.

1 Like

Thanks! Yes, my customers are using Cloudflare. I can tell them to disable proxying and that fixes the problem, but I thought there might be a work-around on the Caddy side for this because I’ve seen so many mentions of Cloudflare in the forums.

Users on the forums that use Cloudflare typically configure the DNS challenge, which allows them to fetch a certificate out-of-band (i.e. not via an HTTP or HTTPS request from the ACME CA to your server), which avoids the issue of having a proxy in front.

But in your case, since you use On-Demand TLS, and you don’t own the credentials to your customer’s Cloudflare accounts, this isn’t an option for you. (Asking your users to give you their Cloudflare credentials would be an insanely bad idea :joy:)

So yeah. You’ll just need to warn them about that. You might be able to do some checks on your backend when they attempt to add their domain to your ask endpoint whitelist to see if their DNS records points to Cloudflare instead of directly to your server, and tell them not to do that.

1 Like

Good idea. I can add a check for this when they initially set up their custom domain with us. Thanks for the clarification.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.