LetsEncrypt Error when Caddy is behind Cloudflare

athenawisdoms · November 4, 2021, 4:20am

1. Caddy version (`caddy version`):

v2.4.5 h1:P1mRs6V2cMcagSPn+NWpD+OEYUYLIf6ecOa48cFGeUg=

2. How I run Caddy:

a. System environment:

Ubuntu 20.04

b. Command:

caddy run --config ~/Caddyfile

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

api.helloworld.net {
	tls admin@helloworld.net

	handle_path /* {
		reverse_proxy localhost:8000
	}

	log {
		output file /root/helloworld/api/logs/access.log {
			roll_size 100mb
			roll_keep 5
			roll_keep_for 720h
		}
	}
}

3. The problem I’m having:

After putting Caddy behind Cloudflare, when Caddy tries to renew LetsEncrypt cert, it is unable to do so, with the following error regarding urn:ietf:params:acme:error:unauthorized:

4. Error messages and/or full log output:

2021/11/04 04:02:16.017 INFO using provided configuration {"config_file": "Caddyfile", "config_adapter": ""}
2021/11/04 04:02:16.021 WARN input is not formatted with 'caddy fmt' {"adapter": "caddyfile", "file": "Caddyfile", "line": 2}
2021/11/04 04:02:16.022 INFO admin admin endpoint started {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["localhost:2019", "[::1]:2019", "127.0.0.1:2019"]}
2021/11/04 04:02:16.022 INFO http server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/11/04 04:02:16.022 INFO http enabling automatic HTTP->HTTPS redirects {"server_name": "srv0"}
2021/11/04 04:02:16.023 INFO http enabling automatic TLS certificate management {"domains": ["api.helloworld.net"]}
2021/11/04 04:02:16.023 INFO autosaved config (load with --resume flag) {"file": "/root/.config/caddy/autosave.json"}
2021/11/04 04:02:16.024 INFO serving initial configuration
2021/11/04 04:02:16.024 INFO tls.obtain acquiring lock {"identifier": "api.helloworld.net"}
2021/11/04 04:02:16.027 INFO tls.obtain lock acquired {"identifier": "api.helloworld.net"}
2021/11/04 04:02:16.030 INFO tls.cache.maintenance started background certificate maintenance {"cache": "0xc000560bd0"}
2021/11/04 04:02:16.030 INFO tls cleaning storage unit {"description": "FileStorage:/root/.local/share/caddy"}
2021/11/04 04:02:16.032 INFO tls finished cleaning storage units
2021/11/04 04:02:16.529 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["api.helloworld.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "admin@helloworld.net"}
2021/11/04 04:02:16.530 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["api.helloworld.net"], "ca": "https://acme-v02.api.letsencrypt.org/directory", "account": "admin@helloworld.net"}
2021/11/04 04:02:16.825 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "api.helloworld.net", "challenge_type": "tls-alpn-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2021/11/04 04:02:17.271 ERROR tls.issuance.acme.acme_client challenge failed {"identifier": "api.helloworld.net", "challenge_type": "tls-alpn-01", "status_code": 403, "problem_type": "urn:ietf:params:acme:error:unauthorized", "error": "Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge"}
2021/11/04 04:02:17.271 ERROR tls.issuance.acme.acme_client validating authorization {"identifier": "api.helloworld.net", "error": "authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Cannot negotiate ALPN protocol \"acme-tls/1\" for tls-alpn-01 challenge", "order": "https://acme-v02.api.letsencrypt.org/acme/order/267181380/36978037040", "attempt": 1, "max_attempts": 3}
2021/11/04 04:02:18.566 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "api.helloworld.net", "challenge_type": "http-01", "ca": "https://acme-v02.api.letsencrypt.org/directory"}
2021/11/04 04:02:19.366 ERROR tls.issuance.acme.acme_client challenge failed {"identifier": "api.helloworld.net", "challenge_type": "http-01", "status_code": 403, "problem_type": "urn:ietf:params:acme:error:unauthorized", "error": "Invalid response from https://api.helloworld.net/.well-known/acme-challenge/V4-ao9FaTaBI9rz8WID4USo-M0IhNnR__HMbYd4kMp0 [2606:4700:3031::ac43:d44a]: \"<!DOCTYPE html>\\n<!--[if lt IE 7]> <html class=\\\"no-js ie6 oldie\\\" lang=\\\"en-US\\\"> <![endif]-->\\n<!--[if IE 7]> <html class=\\\"no-js \""}
2021/11/04 04:02:19.367 ERROR tls.issuance.acme.acme_client validating authorization {"identifier": "api.helloworld.net", "error": "authorization failed: HTTP 403 urn:ietf:params:acme:error:unauthorized - Invalid response from https://api.helloworld.net/.well-known/acme-challenge/V4-ao9FaTaBI9rz8WID4USo-M0IhNnR__HMbYd4kMp0 [2606:4700:3031::ac43:d44a]: \"<!DOCTYPE html>\\n<!--[if lt IE 7]> <html class=\\\"no-js ie6 oldie\\\" lang=\\\"en-US\\\"> <![endif]-->\\n<!--[if IE 7]> <html class=\\\"no-js \"", "order": "https://acme-v02.api.letsencrypt.org/acme/order/267181380/36978042440", "attempt": 2, "max_attempts": 3}
2021/11/04 04:02:20.775 ERROR tls.obtain could not get certificate from issuer {"identifier": "api.helloworld.net", "issuer": "acme-v02.api.letsencrypt.org-directory", "error": "[api.helloworld.net] solving challenges: api.helloworld.net: no solvers available for remaining challenges (configured=[http-01 tls-alpn-01] offered=[http-01 dns-01 tls-alpn-01] remaining=[dns-01]) (order=https://acme-v02.api.letsencrypt.org/acme/order/267181380/36978048610) (ca=https://acme-v02.api.letsencrypt.org/directory)"}
2021/11/04 04:02:21.071 INFO tls.issuance.zerossl generated EAB credentials {"key_id": "ZKpFCor_rMW5q0e12l0D-w"}
2021/11/04 04:02:21.428 INFO tls.issuance.acme waiting on internal rate limiter {"identifiers": ["api.helloworld.net"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "admin@helloworld.net"}
2021/11/04 04:02:21.428 INFO tls.issuance.acme done waiting on internal rate limiter {"identifiers": ["api.helloworld.net"], "ca": "https://acme.zerossl.com/v2/DV90", "account": "admin@helloworld.net"}
2021/11/04 04:02:21.788 INFO tls.issuance.acme.acme_client trying to solve challenge {"identifier": "api.helloworld.net", "challenge_type": "http-01", "ca": "https://acme.zerossl.com/v2/DV90"}

5. What I already tried:

Tried adding :443 to the Caddyfile, but no difference.

api.helloworld.net:443 {
	tls admin@helloworld.net
        ...
}

Also tried using tls internal, but Cloudflare is still giving a “Invalid SSL Certificate” error, with and without :443

api.helloworld.net:443 {
	tls internal
	
	handle_path /* {
		reverse_proxy localhost:8000
	}

	log {
		output file /root/helloworld/api/logs/access.log {
			roll_size 100mb
			roll_keep 5
			roll_keep_for 720h
		}
	}
}

api.helloworld.net {
	tls internal
	
	handle_path /* {
		reverse_proxy localhost:8000
	}

	log {
		output file /root/helloworld/api/logs/access.log {
			roll_size 100mb
			roll_keep 5
			roll_keep_for 720h
		}
	}
}

6. Links to relevant resources:

Caddy and Cloudflare · wonky

Whitestrake · November 4, 2021, 6:27am

Hi @athenawisdoms, I understand the DNS entry in your Cloudflare dashboard is in orange-cloud mode (reverse proxy). What is your site’s SSL/TLS encryption mode?

athenawisdoms · November 4, 2021, 7:00pm

Hey @Whitestrake, I believe its Full (strict).

athenawisdoms · November 4, 2021, 8:36pm

Tried switching to Flexible and the LetsEncrypt errors appear to be gone! Seems that Caddy can now pass the ACME HTTP-01 challenge

But my browser is now giving an error :

ERR_TOO_MANY_REDIRECTS`

athenawisdoms · November 4, 2021, 8:49pm

Switched to Full because I’d like it to be encrypted between Cloudflare and my origin server.

Then configured Caddy to use a self-signed certificate with the tls internal directive

api.helloworld.net {
	tls internal

        ....

To my surprise, when I visit https://api.helloworld.net/some_endpoint, I get the expected response from the server

Whitestrake · November 4, 2021, 11:34pm

I’d still probably advise letting Caddy maintain a trusted certificate, since it costs nothing and precludes self-signed MITM attacks, but I’m glad to hear you’ve got things working!

By way of an explanation: Full (strict) only connects via HTTPS on port 443 and requires Caddy provide a valid certificate (which it can’t do yet since it still needs to solve an ACME challenge and acquire one). You can use this mode with Caddy, just not the first time you run it on each site - Caddy renews certificates early in the tail end of validity so HTTPS is set-and-forget once you’ve got it running in this mode.

Flexible only connects via HTTP on port 80, but provides the client with valid HTTPS. Either this mode or Off (not secure) mode is required the first time you add your site to Caddy (unless you use DNS validation), because Caddy can’t solve TLS-ALPN challenges behind Cloudflare (since Cloudflare terminates HTTPS), so it requires HTTP challenges to solve.

Once the solving is complete for the first run, though, Caddy will start trying to redirect the client up HTTP->S. This is the part that produces a redirect loop; the client is already trying to connect on HTTPS, but Cloudflare is connecting on HTTP only, hence continuous redirection. So, you’ll need to swap it back off this mode after the challenge is first solved.

TL;DR: The ideal scenario is to use Flexible to solve the ACME challenge the first time, then go to Full (strict) afterwards as Caddy can maintain a certificate in Full (strict) mode, but can’t acquire a fresh one.

Alternately, leave the site in Full (strict) mode but grey-cloud your website for the first run, then orange-cloud it after Caddy has acquired a certificate, for a similar result.

athenawisdoms · November 5, 2021, 3:48am

Thanks for the explanation!!

I think I did as you have described in your “TLDR ideal scenario”, but then it stopped working after some months.

Is this because when in Full (strict) mode, it fails to renew the LetsEncrypt certificate (“can’t acquire a fresh one.” like you have mentioned)?

Whitestrake · November 5, 2021, 4:05am

The two more common ACME challenges are HTTP (I request a document with a specific response; if you provide the correct HTTP response body, I give you a cert) - and TLS-ALPN (I connect over HTTPS; while we are negotiating the TLS protocol, I challenge and you respond during that negotiation; if you provide the correct response, I give you a cert).

The latter can’t ever happen behind Cloudflare because the public doesn’t ever negotiate TLS directly with Caddy; the ACME server will negotiate TLS with Cloudflare (who can then, depending on configuration, negotiate TLS separately with you), who doesn’t know the challenge response.

So TLS-ALPN is out; HTTP is our only option here other than configuring DNS validation. HTTP challenges work fine behind a reverse proxy. Initially that has to happen over HTTP (or self-signed HTTPS) because Caddy can’t provide a certificate it doesn’t have yet.

HOWEVER, you can still solve a HTTP challenge over HTTPS! All it’s doing is requesting a document that only Caddy knows the right response for. Basically, HTTPS is just HTTP over TLS. So once Caddy has a valid cert, it can keep renewing that cert indefinitely as long as it does so within the validity window of the previous cert. Since Caddy typically tries to renew inside the last 30 days of validity, it should always have plenty of time to do that.

So, to answer your question, it might stop working if you stop it from renewing its certificate long enough for the certificate to expire; but it shouldn’t stop working after you’ve got it going, not without something else going wrong.

system · December 4, 2021, 4:20am

This topic was automatically closed after 30 days. New replies are no longer allowed.