Cloudflare 522 errors - site inaccessible

What I’m trying to do:

Host a .index and reverse proxy a Jellyfin container

My domain is inaccessible and returns Cloudflare’s 522 errors when I try to load it up. I figure if I manage to access my .index, the reverse proxy will also work. I have had all this working before, but I have seemingly broken in when I did a fresh install of OpenMediaVault.

My setup:

OS: OpenMediaVault 4 running on a ThinkServer TS130, containers are run in a Docker environment.

Steps I have taken

1. Changed DNS servers in OpenMediaVault and my Asus-Merlin router to 1.1.1.1 and 8.8.8.8

2. Disabled firewall in Asus-Merlin router

2.1 Forwarded ports 443 and 80 in Asus-Merlin router and on OMV machine.

3. Created an .index with resources at /path/www/goph

4. Setup my domain with Cloudflare

curl ifconfig.me
returns the same IP as in the top-level record

4.1 Configured SSL, Caching and Page Rules following https://github.com/PGBlitz/PGBlitz.com/wiki/CloudFlare

5. Setup a Cloudflare DDNS container (joshuaavalon/cloudflare-ddns) to update my IP to the Cloudflare records.

docker run
-d
–restart unless-stopped
–name=cloudflare-ddns
-e ZONE=goph.no
-e HOST=goph.no
-e EMAIL=censored@gmail.com
-e API=censored
-e TTL=1
-e PROXY=true
joshuaavalon/cloudflare-ddns

6. Setup a Caddy container (abiosoft/caddy)

docker run
-d
–restart unless-stopped
–name=caddy
–cap-add=NET_ADMIN
-e CLOUDFLARE_EMAIL=censored@gmail.com
-e CLOUDFLARE_API_KEY=censored
-e PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
-e ACME_AGREE=true
-v /sharedfolders/appdata/caddy/.caddy:/root/.caddy:rw
-v /sharedfolders/appdata/caddy/Caddyfile:/etc/Caddyfile:rw
-v /srv:/srv:rw \ #(this is the path to my .index)
-p 2015:2015/tcp
-p 443:443/tcp
-p 80:80/tcp
abiosoft/caddy

6.1 Configuring a Caddyfile

goph.no {
root /srv/dev-disk-by-label-fsd/www/goph
tls {
dns cloudflare
}
}

6.2 Running the Caddy container, logs returned are (logs edited - removed links according to new user link limitation on the forum)

Activating privacy features… 2019/09/23 10:29:17 [INFO][cache:0xc0001ac320] Started certificate maintenance routine

Your sites will be served over HTTPS automatically using Let’s Encrypt.
By continuing, you agree to the Let’s Encrypt Subscriber Agreement at:
https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf
Please enter your email address to signify agreement and to be notified
in case of issues. You can leave it blank, but we don’t recommend it.
Email address: 2019/09/23 10:29:19 [INFO][goph.no] Obtain certificate
2019/09/23 10:29:19 [INFO] [goph.no] acme: Obtaining bundled SAN certificate
2019/09/23 10:29:21 [INFO] [goph.no] AuthURL: https (://) acme-v02.api.letsencrypt (dot) org/acme/authz-v3/468092978
2019/09/23 10:29:21 [INFO] [goph.no] acme: Could not find solver for: tls-alpn-01
2019/09/23 10:29:21 [INFO] [goph.no] acme: Could not find solver for: http-01
2019/09/23 10:29:21 [INFO] [goph.no] acme: use dns-01 solver
2019/09/23 10:29:21 [INFO] [goph.no] acme: Preparing to solve DNS-01
2019/09/23 10:29:21 [INFO] cloudflare: new record for goph.no, ID bea735a37dad2a1b17e8c2f38573ad65
2019/09/23 10:29:21 [INFO] [goph.no] acme: Trying to solve DNS-01
2019/09/23 10:29:21 [INFO] [goph.no] acme: Checking DNS record propagation using [192.168.1.1:53 1.1.1.1:53 8.8.8.8:53]
2019/09/23 10:29:21 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2019/09/23 10:29:21 [INFO] [goph.no] acme: Waiting for DNS record propagation.
2019/09/23 10:29:24 [INFO] [goph.no] acme: Waiting for DNS record propagation.
2019/09/23 10:29:26 [INFO] [goph.no] acme: Waiting for DNS record propagation.
2019/09/23 10:29:29 [INFO] [goph.no] The server validated our request
2019/09/23 10:29:29 [INFO] [goph.no] acme: Cleaning DNS-01 challenge
2019/09/23 10:29:30 [INFO] [goph.no] acme: Validations succeeded; requesting certificates
2019/09/23 10:29:32 [INFO] [goph.no] Server responded with a certificate.
2019/09/23 10:29:32 [INFO][jf.goph.no] Obtain certificate
2019/09/23 10:29:32 [INFO] [jf.goph.no] acme: Obtaining bundled SAN certificate
2019/09/23 10:29:35 [INFO] [jf.goph.no] AuthURL: https (://) acme-v02.api.letsencrypt (dot) org/acme/authz-v3/468095025
2019/09/23 10:29:35 [INFO] [jf.goph.no] acme: Could not find solver for: tls-alpn-01
2019/09/23 10:29:35 [INFO] [jf.goph.no] acme: Could not find solver for: http-01
2019/09/23 10:29:35 [INFO] [jf.goph.no] acme: use dns-01 solver
2019/09/23 10:29:35 [INFO] [jf.goph.no] acme: Preparing to solve DNS-01
2019/09/23 10:29:36 [INFO] cloudflare: new record for jf.goph.no, ID 2af648ebaee4c84bc6c02c9a185a8754
2019/09/23 10:29:36 [INFO] [jf.goph.no] acme: Trying to solve DNS-01
2019/09/23 10:29:36 [INFO] [jf.goph.no] acme: Checking DNS record propagation using [192.168.1.1:53 1.1.1.1:53 8.8.8.8:53]
2019/09/23 10:29:36 [INFO] Wait for propagation [timeout: 2m0s, interval: 2s]
2019/09/23 10:29:36 [INFO] [jf.goph.no] acme: Waiting for DNS record propagation.
2019/09/23 10:29:38 [INFO] [jf.goph.no] acme: Waiting for DNS record propagation.
2019/09/23 10:29:40 [INFO] [jf.goph.no] acme: Waiting for DNS record propagation.
2019/09/23 10:29:42 [INFO] [jf.goph.no] acme: Waiting for DNS record propagation.
2019/09/23 10:29:47 [INFO] [jf.goph.no] The server validated our request
2019/09/23 10:29:47 [INFO] [jf.goph.no] acme: Cleaning DNS-01 challenge
2019/09/23 10:29:47 [INFO] [jf.goph.no] acme: Validations succeeded; requesting certificates
2019/09/23 10:29:51 [INFO] [jf.goph.no] Server responded with a certificate.
done.

Serving HTTPS on port 443
https (://) goph (dot) no
https (://) jf.goph (dot) no

2019/09/23 10:29:51 [INFO] Serving https://goph (dot) no
2019/09/23 10:29:51 [INFO] Serving https://jf (dot) goph (dot) no

Serving HTTP on port 80
http (://) goph (dot) no
http (://) jf.goph (dot) no

2019/09/23 10:29:51 [INFO] Serving http://goph.no
2019/09/23 10:29:51 [INFO] Serving http://jf.goph.no
2019/09/23 10:29:52 [INFO] Sending telemetry: success

7. Trying to access domain in a browser (tested both inside and outside network)
Returns

Error 522 Ray ID: 51ac11743e28769a • 2019-09-23 11:10:15 UTC

Connection timed out

EDIT: I have removed and added the Caddy container too many times and have been rate-limited by LetsEncrypt – I’ll try again next week. Please, look over my post and and give me some feedback, as I feel like I have tried every possible thing to fix this but to no avail.

Oof. I haven’t had time to look into the issue (traveling today) but: Next time, please use Let’s Encrypt’s staging endpoint when debugging/developing: https://caddyserver.com/docs/automatic-https#testing - and you will avoid rate limits.

Gotcha! Thanks for letting me know :blush:

It’s not like I’ll be able to do something for a little while anyway! I am looking forward to some of your insight though :smile:

1 Like

This is a networking problem, for sure.

Either DDNS isn’t being updated properly, OR the firewall is not configured to allow traffic on HTTP(S) ports, OR your router is not forwarding HTTP(S) ports correctly to the Caddy host, OR some combination of the above.

On the Discord server, I advised running curl -ILH "Host: <your domain name>" <your external IP> to test your origin directly, and you replied that the result was:

curl: (7) Failed to connect to 109.deleted.deleted.140 port 80: Connection timed out

That means that along the path to your IP, something isn’t responding. Usually a firewall.

You also mentioned on Discord that when you tried to access your site over HTTPS on port 443, you got the error:

400 Bad Request The plain HTTP request was sent to HTTPS port

I’m gonna be honest, I’ve got no clue how that happened. You said you connected to https://domain.com:443/, so you sent a HTTPS request, and somewhere along the way it got translated to HTTP and sent to a HTTPS port. Cloudflare doesn’t fail in this manner when misconfigured and combined with Caddy - it loops redirects. There’s something else in the way.

Yes, this must be network related – 100% agree.

The weird thing though is that my router settings has been completely unchanged since I ran Caddy successfully on the previous install of OMV.

Caddy is simple – have a decent Caddyfile, the right ports open and you’re good to go.

Is there any router related settings which could be wise to toggle/disable/change when you’re running Caddy?

I’m going to keep experimenting until I hit a successful curl -ILH "Host: <your domain name>" <your external IP> command.

Well, frankly, no. I can’t think of anything else. As you say, it should be quite simple.

Did you end up re-enabling the firewall on your router, by the way?

Start shedding complexity wherever you can, e.g. grey-cloud the DNS record.

Solved it!

My ISP had given me a CGN-IP which messed up my reachability.

I talked to my ISP and purchased a public IP lease – everything works flawlessly.

And here I was considering reverting to NGINX.

Thanks for the help Matthew and Matt!

2 Likes