Getting Error 526 (Invalid SSL Certificate) using Caddy with Cloudflare TLS. What's going wrong here?


(Recklessnl) #1

Hi all,

So I am in the process of setting up a Caddy reverse proxy server, in an LXC container (running clean standard Debian 9.7.1 stable, nothing else on there). Latest Caddy version as of yesterday, only used the Cloudflare TLS DNS plugin additional to the main Caddy download. This is my Caddyfile:

test.domain.com {
	proxy / 192.168.1.23:1234 {
	transparant
	}
	tls {
	dns cloudflare
	}
}

I own domain.com and in Cloudflare I have done the following:

  • A record for domain.com (points to my local server IP) in Cloudflare
  • CNAME for test (is an alias of domain.com) in Cloudflare
  • In Crypto settings on Cloudflare, I made sure to put the SSL setting to Full (strict).
  • All the A and CNAME records in Cloudflare are orange (or have the orange cloud activated), which means the extra Cloudflare protections are enabled.
  • I have also forwarded port 80 and 443 on my router to the Caddy server IP.

I installed Caddy and since I use the TLS DNS Cloudflare way of getting a cert, this is how I run my caddy command:

CLOUDFLARE_EMAIL=email@domain.com CLOUDFLARE_API_KEY=my_api_key caddy

This yields the following result (swapped out my actual domain and scrambled the unique IDs for privacy reasons):

root@caddy:~# CLOUDFLARE_EMAIL=email@domain.com CLOUDFLARE_API_KEY=my_api_key caddy
Activating privacy features... 2019/03/04 09:30:37 [INFO] [test.domain.com] acme: Obtaining bundled SAN certificate
2019/03/04 09:30:38 [INFO] [test.domain.com] AuthURL: https://acme-v02.api.letsencrypt.org/acme/authz/-DoDVgmCQhf23l2if7KfVh3Fw9fzwOsm4xlLMaoLQZQ
2019/03/04 09:30:38 [INFO] [test.domain.com] acme: Could not find solver for: tls-alpn-01
2019/03/04 09:30:38 [INFO] [test.domain.com] acme: Could not find solver for: http-01
2019/03/04 09:30:38 [INFO] [test.domain.com] acme: use dns-01 solver
2019/03/04 09:30:38 [INFO] [test.domain.com] acme: Preparing to solve DNS-01
2019/03/04 09:30:38 [INFO] cloudflare: new record for test.domain.com, ID 0dofka1867ba2cjdk3958f6ee47aab73
2019/03/04 09:30:38 [INFO] [test.domain.com] acme: Trying to solve DNS-01
2019/03/04 09:30:38 [INFO] [test.domain.com] acme: Checking DNS record propagation using [192.168.1.10:53]
2019/03/04 09:30:38 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2019/03/04 09:30:38 [INFO] [test.domain.com] acme: Waiting for DNS record propagation.
2019/03/04 09:30:40 [INFO] [test.domain.com] acme: Waiting for DNS record propagation.
2019/03/04 09:30:42 [INFO] [test.domain.com] acme: Waiting for DNS record propagation.
2019/03/04 09:30:44 [INFO] [test.domain.com] acme: Waiting for DNS record propagation.
2019/03/04 09:30:46 [INFO] [test.domain.com] acme: Waiting for DNS record propagation.
2019/03/04 09:30:49 [INFO] [test.domain.com] acme: Waiting for DNS record propagation.
2019/03/04 09:30:56 [INFO] [test.domain.com] The server validated our request
2019/03/04 09:30:56 [INFO] [test.domain.com] acme: Cleaning DNS-01 challenge
2019/03/04 09:30:57 [INFO] [test.domain.com] acme: Validations succeeded; requesting certificates
2019/03/04 09:30:58 [INFO] [test.domain.com] Server responded with a certificate.
done.
https://test.domain.com
http://test.domain.com
WARNING: File descriptor limit 1024 is too low for production servers. At least 8192 is recommended. Fix with `ulimit -n 8192`.

This all looks good, but then when I actually try the website I get Error 526 on Cloudflare. This is what Cloudflare says about Error 526: https://support.cloudflare.com/hc/en-us/articles/200721975-Error-526-Invalid-SSL-certificate

Any ideas on what is going wrong and how I can make it work? Thanks for the help.


(Matthew Fay) #2

Hi @recklessnl, welcome to the Caddy community.

domain.com belongs to another entity. You may use example.com for demonstrative purposes instead.

Can you elaborate on how you run Caddy (what parameters, etc)? (Edit: Ignore that, you already specified.) You might also check that:

  • The DNS record in your Cloudflare dashboard points to the right origin server IP address
  • A direct request (such as via curl) to your origin server returns a good result (i.e. valid certificate)

(Recklessnl) #3

Hi Matthew,

Can you elaborate on how you run Caddy (what parameters, etc)?

Everything I’ve done is described in my original post, I haven’t changed any other settings. I installed Caddy using the bash install script (which I got straight from the main Caddy website). I haven’t changed anything other than I listed above. I have the Caddyfile in the home folder, I run as root user since it’s just a simple LXC container.

The DNS record in your Cloudflare dashboard points to the right origin server IP address

The A records of example.com indeed point to my server IP on the cloudflare webpage.

A direct request (such as via curl ) to your origin server returns a good result (i.e. valid certificate)

What command should I run exactly? If I do curl https://test.example.com I just get the source code of the Cloudflare error site, not sure what command I should run.


(Matthew Fay) #4

You need to override the DNS resolution that curl usually does in order to point it manually at the origin itself. The --resolve flag lets you manually specify an IP address to use for a port and hostname on a given request:

curl -IL https://test.example.com --resolve test.example.com:443:0.0.0.0


(Recklessnl) #5

Running your command gives the following output:

HTTP/2 405 
allow: GET
content-type: text/plain; charset=utf-8
date: Tue, 05 Mar 2019 01:03:23 GMT
server: Caddy
server: Python/3.7 aiohttp/3.5.4
content-length: 23

(Matthew Fay) #6

You could slightly modify from curl -IL to curl -ILX GET to fix that status 405, but suffice to say it’s connecting over HTTPS and not running into any validation issues, which is good.

So Caddy’s doing its job, which means it’s something wrong on the Cloudflare side of things. Try relaxing the SSL requirement from Strict to Full, make another connection attempt, and see what you get.


(Recklessnl) #7

I have a feeling this issue might have to do with my old Synology box. I initially had a reverse proxy set-up there that didn’t work very well - and it handled a certificate for example.com (but NOT the CNAME of test.example.com). Perhaps that is what messing with it - because when I switch up Cloudflare to Full instead of Full (strict), go to test.example.com, I get a connection timeout error and I see that the URL port gets redirected to 5001, which is a Synology port. That is really strange…

Due to that 5001 port being added, I have a feeling that could be a factor here. Is there a way to reset the certficated? I can easily delete them from my Synology reverse proxy GUI, should I do that?

Additionally, is it OK to have port 80 forwarded to both my Synology box and also the new Caddy LXC container? That’s what I currently do (port 443 only goes to the Caddy server).

Is there a way to reset the certs and have my domain start from scratch with the certs in order to get everything working?


(Matthew Fay) #8

In short: Nope.

How can the the edge firewall decide which host to send those packets to? It has to send it to one. It makes no sense to duplicate/mirror the packets. So it will always send them to the first port forward on the list (I’m surprised it let you add the same port forward to multiple hosts).

My strong recommendation is to forward only to Caddy, then have Caddy reverse proxy your Synology box, if appropriate. Caddy’s very good at what it does, very simple, and makes accommodating less-than-simple apps like Synology’s GUI very straightforward.

With all that said, if port 443 only goes to Caddy already, I’m not sure how you ended up with a redirect to Synology unless Caddy or the Python upstream server issued that redirect. I’d definitely consider wiping the port forward list out and set up entries from scratch to ensure there’s no weirdness going on.


(Recklessnl) #9

Aaaand that’s what the issue was! Port 443 was forwarded to both Caddy and the Synology box. Removing the Synology port forward was all it took to fix it. Yay! Thanks so much for your help.