"525 SSL Handshake Failed" when using let's encrpyt auto acquired certificate with cloudflare

nauxihziq · January 12, 2023, 2:25am

I am trying to build a proxy service using forward_proxy, reverse_proxy, and the Cloudflare DNS modules. I planned to migrate my old WebDAV service (on another VPS) as well, but I eventually removed the block (and ran) to make it clear to figure out what was wrong.

1. Output of `caddy version`:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

2. How I run Caddy:

~/go/bin/xcaddy build \
--with github.com/mholt/caddy-webdav \
--with github.com/caddyserver/forwardproxy=$PWD/forwardproxy \
--with github.com/caddy-dns/cloudflare

a. System environment:

OS: Debian GNU/Linux bookworm/sid

b. Command:

systemctl start caddy

or

caddy run --config /etc/caddy/config.json

c. Service/unit/compose file:

caddy.service

[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target

[Service]
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/config.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

{
	"admin": {
		"disabled": true
	},
	"apps": {
		"http": {
			"servers": {
				"static_proxy":{
					"listen": [":4433"],
					"routes": [{
						"handle": [{
						"handler": "forward_proxy",
						"hide_ip": true,
						"hide_via": true,
						"auth_user_deprecated": "****",
						"auth_pass_deprecated": "****",
						"probe_resistance": {}
						}]
					}, 
					{
						"match": [{"host": ["phil.****.com"]}],
						"handle": [{
						"handler": "file_server",
						"root": "/var/www/html/phil.****.com/"
						}],
						"terminal": true
					}],
					"tls_connection_policies": [{
						"match": {"sni": ["phil.zhixuanqi.com"]}
					}]
				},
				"forwardproxy": {
					"listen": [":443"],
					"routes": [{
						"match": [{"host": ["bjut.****.com"]}],
						"handle": [{
							"handler": "forward_proxy",
							"hide_ip": true,
              				"hide_via": true,
              				"auth_user_deprecated": "****",
              				"auth_pass_deprecated": "****",
              				"probe_resistance": {}
						},
						{
							"handler": "reverse_proxy",
							"transport": {
								"protocol": "http",
								"tls": {}
							},
							"upstreams": [{
								"dial": "https://english.****.edu.cn/Identity/Alumni/Introduction.htm"
							}],
							"headers": {
								"request": {
									"set": {
										"Host": ["{http.reverse_proxy.upstream.hostport}"],
										"X-Forwarded-Host": ["{http.request.host}"],
										"User-Agent": ["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) "]
									}
								}
							}
						}],
						"terminal": true
					}],
					"tls_connection_policies": [{
						"match": {
							"sni": ["bjut.****.com"]
						}
					}]
				}
			}
		},
		"tls": {
			"automation": {
				"policies": [{
					"subjects": ["bjut.****.com", "phil.****.com"],
					"issuers": [{
						"module": "acme",
						"email": "zhixuanqi@****.com",
						"challenges": {
							"dns": {
								"provider": {
									"name": "cloudflare",
									"api_token": "****"
								}
							}
						}
					}
				]
				}]
			}
		}
	}	
}

[Updated] I have updated this config.json file again to see whether the static site with forward_proxy would work but it failed as well.

3. The problem I’m having:

I could not access my server by the domain I specified. Caddy log outputs suggested that the server is only listening to the HTTP port, which is not how I configured it. According to the nice Cloudflare page, the host was not responding. I guess that the SSL certificate is not issued for some reason (probably because I didn’t properly specify the certification process).

I am a newbie who spent all night writing and modifying the JSON file and looking at the official structure guide. Feeling really frustrated, I wonder if there is a more efficient way to configure the caddy to behave in a proper way. Sorry in advance for my broken English, and thanks for your help!

[updated] when I am trying to visit the static site phil.****.com, I got 525 SSL Handshake Fail from cloudflare, while when I try to access the reverse proxy site bjut.****.com, the output log goes crazy. (see the latest appended error message)

4. Error messages and/or full log output:

2023/01/12 01:33:45.439	WARN	http	server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server	{"server_name": "forwardproxy", "http_port": 80}
2023/01/12 01:33:45.439	INFO	http.log	server running	{"name": "forwardproxy", "protocols": ["h1", "h2", "h3"]}
2023/01/12 01:33:45.440	INFO	autosaved config (load with --resume flag)	{"file": "/root/.config/caddy/autosave.json"}
2023/01/12 01:33:45.440	INFO	serving initial configuration
2023/01/12 01:33:45.440	INFO	tls	cleaning storage unit	{"description": "FileStorage:/root/.local/share/caddy"}
2023/01/12 01:33:45.440	INFO	tls	finished cleaning storage units
2023/01/12 01:33:45.440	INFO	tls.cache.maintenance	started background certificate maintenance	{"cache": "0xc00013ae00"}

I ran it again after a while and the output goes like:

Jan 12 02:11:44 localhost caddy[3856266]: {"level":"info","ts":1673489504.9973767,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"forwardproxy"}
Jan 12 02:11:44 localhost caddy[3856266]: {"level":"info","ts":1673489504.998093,"logger":"http","msg":"enabling HTTP/3 listener","addr":":443"}
Jan 12 02:11:44 localhost caddy[3856266]: {"level":"info","ts":1673489504.9987707,"msg":"failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/qui>
Jan 12 02:11:44 localhost caddy[3856266]: {"level":"info","ts":1673489504.9991124,"logger":"http.log","msg":"server running","name":"forwardproxy","protocols":["h1","h2","h3"]}
Jan 12 02:11:44 localhost caddy[3856266]: {"level":"info","ts":1673489504.99928,"logger":"http.log","msg":"server running","name":"remaining_auto_https_redirects","protocols":["h1","h2","h3"]}
Jan 12 02:11:44 localhost caddy[3856266]: {"level":"info","ts":1673489504.999506,"msg":"autosaved config (load with --resume flag)","file":"/var/lib/caddy/.config/caddy/autosave.json"}
Jan 12 02:11:44 localhost caddy[3856266]: {"level":"info","ts":1673489504.9996116,"msg":"serving initial configuration"}
Jan 12 02:11:45 localhost caddy[3856266]: {"level":"info","ts":1673489504.999951,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc0000acd90"}
Jan 12 02:11:45 localhost caddy[3856266]: {"level":"info","ts":1673489504.9999971,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/var/lib/caddy/.local/share/caddy"}
Jan 12 02:11:45 localhost caddy[3856266]: {"level":"info","ts":1673489505.0000088,"logger":"tls","msg":"finished cleaning storage units"}

[Updated] The output when I tried to visit the reverse proxy site bjut.****.com

2023/01/12 04:25:19.661	ERROR	http.log.error	dial https:: unknown network https:	{"request": {"remote_ip": "****", "remote_port": "51042", "proto": "HTTP/2.0", "method": "GET", "host": "bjut.****.com", "uri": "/", "headers": {"Sec-Fetch-User": ["?1"], "Cf-Ipcountry": ["JP"], "Dnt": ["1"], "Accept-Language": ["en-US,en;q=0.5"], "Sec-Fetch-Mode": ["navigate"], "X-Forwarded-For": ["23.106.140.60"], "Cf-Visitor": ["{\"scheme\":\"https\"}"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8"], "Upgrade-Insecure-Requests": ["1"], "Sec-Fetch-Dest": ["document"], "Cf-Connecting-Ip": ["23.106.140.60"], "Cdn-Loop": ["cloudflare"], "User-Agent": ["Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:108.0) Gecko/20100101 Firefox/108.0"], "X-Forwarded-Proto": ["https"], "Sec-Fetch-Site": ["none"], "Accept-Encoding": ["gzip"], "Cf-Ray": ["788320a92d4f0a9a-ATL"]}, "tls": {"resumed": false, "version": 772, "cipher_suite": 4865, "proto": "h2", "server_name": "bjut.****.com"}}, "duration": 0.000249156, "status": 502, "err_id": "3sgee6i1n", "err_trace": "reverseproxy.statusError (reverseproxy.go:1272)"}

5. What I already tried:

I have checked the listening port previously and the output is consistent with the newer version of caddy output

Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 127.0.0.1:48609         0.0.0.0:*               LISTEN      3847612/python
tcp        0      0 127.0.0.1:41955         0.0.0.0:*               LISTEN      40802/python
tcp        0      0 127.0.0.1:44473         0.0.0.0:*               LISTEN      3847612/python
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      557/sshd: /usr/sbin
tcp        0      0 127.0.0.1:60761         0.0.0.0:*               LISTEN      40802/python
tcp        0      0 127.0.0.1:45409         0.0.0.0:*               LISTEN      40802/python
tcp        0      0 0.0.0.0:8888            0.0.0.0:*               LISTEN      22673/python
tcp        0      0 0.0.0.0:8889            0.0.0.0:*               LISTEN      38211/python
tcp        0      0 127.0.0.1:46785         0.0.0.0:*               LISTEN      40802/python
tcp        0      0 127.0.0.1:39619         0.0.0.0:*               LISTEN      40802/python
tcp        0      0 127.0.0.1:52451         0.0.0.0:*               LISTEN      3847612/python
tcp        0      0 127.0.0.1:32999         0.0.0.0:*               LISTEN      3847612/python
tcp        0      0 127.0.0.1:40081         0.0.0.0:*               LISTEN      40802/python
tcp        0      0 127.0.0.1:36999         0.0.0.0:*               LISTEN      3847612/python
tcp        0      0 127.0.0.1:40537         0.0.0.0:*               LISTEN      3847612/python
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      42038/exim4
tcp6       0      0 :::80                   :::*                    LISTEN      3856266/caddy
tcp6       0      0 :::22                   :::*                    LISTEN      557/sshd: /usr/sbin
tcp6       0      0 :::8888                 :::*                    LISTEN      22673/python
tcp6       0      0 :::8889                 :::*                    LISTEN      38211/python
tcp6       0      0 ::1:25                  :::*                    LISTEN      42038/exim4
tcp6       0      0 :::443                  :::*                    LISTEN      3856266/caddy

6. Links to relevant resources:

https://www.oilandfish.com/posts/naiveproxy-caddy-2.html

francislavoie · January 12, 2023, 10:19am

First, a couple comments on things I noticed in the config:

This won’t work – reverse_proxy upstreams cannot take a scheme or path, it’s only a dial address so it should only be a hostname/IP and port. The tls config in the transport is what enables HTTPS. If you need to rewrite to a specific path, use the rewrite handler.

You don’t need this, Caddy will add this header by default.

Yeah, that’s what I was referencing above, the dial address is bad so that’s what happens.

Are you still having a 525 error from CloudFlare? I’m not seeing any evidence of TLS issuance issues from your logs, it looks like the cert was correctly issued from a previous run, since there’s no complaints from the tls app in your logs.

nauxihziq · January 12, 2023, 12:52pm

I have switched the dial with a valid network address and the reverse_site worked just fine!

But the forward_proxy handler from the reverse_poxy server seemed not behaving properly as I tried to connect to it using a proxy client on my PC.

Moreover, when I was trying to visit the static site (server static_proxy), I got the 525 SSL Handshake Failed message again.

I tried to access this site by visiting it’s <ip_addr>:<port> directly and I got
Secure Connection Failed: Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

And I tried the same thing with the reverse_proxy, it was failed the same way
Secure Connection Failed: Error code: SSL_ERROR_INTERNAL_ERROR_ALERT

I think it has something to do with the SSL certificates? I am currently in strict mode with all stated domains with cloudflare.

Thank you again for your help!

nauxihziq · January 12, 2023, 6:08pm

I later managed to host the static site with forward_proxy in another JSON configuration, but the proxy client never worked. I later find out that it was because the protocol I used, naiveproxy does not support CDNs (see). Therefore the server was actually functional.

system · February 11, 2023, 2:26am

This topic was automatically closed after 30 days. New replies are no longer allowed.