Caddy through Docker + Synology - Failed to receive handshake

Hankerdoodle · December 13, 2022, 7:32pm

1. Output of `caddy version`:

v2.6.2

2. How I run Caddy:

a. System environment:

Synology NAS and Docker

b. Command:

caddy run --config /etc/caddy/Caddyfile --adapter caddyfile

Default container command afaik

c. Service/unit/compose file:

Running this all through Portainer so not sure if I get extract a Dockerfile from there?

d. My complete Caddy config:

files.wichelo.nl {
	reverse_proxy 192.168.2.11:7001
}

3. The problem I’m having:

Browser not showing the desired output. curl output below:

curl -v https://files.wichelo.nl/
*   Trying 37.251.33.34:443...
* Connected to files.wichelo.nl (37.251.33.34) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* schannel: failed to receive handshake, SSL/TLS connection failed
* Closing connection 0
curl: (35) schannel: failed to receive handshake, SSL/TLS connection failed

4. Error messages and/or full log output:

INF ts=1670955945.4331179 msg=using provided configuration config_file=/etc/caddy/Caddyfile config_adapter=caddyfile

INF ts=1670955945.435062 logger=admin msg=admin endpoint started address=localhost:2019 enforce_origin=false origins=["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]

INF ts=1670955945.4352381 logger=http msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443

INF ts=1670955945.435255 logger=http msg=enabling automatic HTTP->HTTPS redirects server_name=srv0

INF ts=1670955945.435284 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc00064a690

INF ts=1670955945.4355686 logger=http msg=enabling HTTP/3 listener addr=:443

INF ts=1670955945.4355915 logger=tls msg=cleaning storage unit description=FileStorage:/data/caddy

INF ts=1670955945.4356382 msg=failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.

INF ts=1670955945.4357493 logger=http.log msg=server running name=srv0 protocols=["h1","h2","h3"]

INF ts=1670955945.4357836 logger=http.log msg=server running name=remaining_auto_https_redirects protocols=["h1","h2","h3"]

INF ts=1670955945.4357908 logger=http msg=enabling automatic TLS certificate management domains=["files.wichelo.nl"]

INF ts=1670955945.436389 logger=tls msg=finished cleaning storage units

INF ts=1670955945.4367976 msg=autosaved config (load with --resume flag) file=/config/caddy/autosave.json

INF ts=1670955945.4368105 msg=serving initial configuration

5. What I already tried:

So far I have routed the port as follows:

I’ve got a DDNS setup through Google that seems to work fine.
After setting up the port forwarding as shown above, the logs no longer showed any errors regarding its connection (previously I had forwarded the wrong ports). The desired interface to be shown through the URL (Synology NAS File Station Login page) was however still not showing up. From here I understood the certificates need to be grabbed and installed on the NAS, but that also did not remedy the issue.

There’s probably something glaringly obvious I am missing.

6. Links to relevant resources:

francislavoie · December 16, 2022, 6:12am

I don’t think you’re actually reaching Caddy. I think some broken (outdated) HTTPS server is intercepting the request. You’ll need to dig deeper on your networking to make sure it’s all wired up correctly.

Hankerdoodle · December 19, 2022, 5:14pm

Thanks for your reply. Is there any way to properly debug/trace what is interfering? My network is quite simple as in apart from a router, some accesspoints and a NAS nothing should intercept the request.

francislavoie · December 19, 2022, 5:27pm

You can add the debug global option to your Caddyfile, which will output more verbose logs. Add this at the top of your Caddyfile:

{
	debug
}

Aside from that, if it’s a networking issue, then it’s not directly related to Caddy, and there’s not much we can do to support you there.

Hankerdoodle · December 19, 2022, 7:58pm

Apart from a few extra lines, no extra info on the why of the problem.

INF ts=1671475594.392978 msg=using provided configuration config_file=/etc/caddy/Caddyfile config_adapter=caddyfile

INF ts=1671475594.395034 logger=admin msg=admin endpoint started address=localhost:2019 enforce_origin=false origins=["//localhost:2019","//[::1]:2019","//127.0.0.1:2019"]

INF ts=1671475594.3951828 logger=http msg=server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS server_name=srv0 https_port=443

INF ts=1671475594.3951979 logger=http msg=enabling automatic HTTP->HTTPS redirects server_name=srv0

INF ts=1671475594.3952932 logger=tls.cache.maintenance msg=started background certificate maintenance cache=0xc00061c700

INF ts=1671475594.3954873 logger=http msg=enabling HTTP/3 listener addr=:443

INF ts=1671475594.3955286 logger=tls msg=cleaning storage unit description=FileStorage:/data/caddy

INF ts=1671475594.3955503 msg=failed to sufficiently increase receive buffer size (was: 208 kiB, wanted: 2048 kiB, got: 416 kiB). See https://github.com/lucas-clemente/quic-go/wiki/UDP-Receive-Buffer-Size for details.

DBG ts=1671475594.395689 logger=http msg=starting server loop address=[::]:443 tls=true http3=true

INF ts=1671475594.3957078 logger=http.log msg=server running name=srv0 protocols=["h1","h2","h3"]

DBG ts=1671475594.3957374 logger=http msg=starting server loop address=[::]:80 tls=false http3=false

INF ts=1671475594.3957446 logger=http.log msg=server running name=remaining_auto_https_redirects protocols=["h1","h2","h3"]

INF ts=1671475594.3957493 logger=http msg=enabling automatic TLS certificate management domains=["files.wichelo.nl"]

DBG ts=1671475594.396182 logger=tls msg=loading managed certificate domain=files.wichelo.nl expiration=1678543247 issuer_key=acme-v02.api.letsencrypt.org-directory storage=FileStorage:/data/caddy

INF ts=1671475594.396404 logger=tls msg=finished cleaning storage units

DBG ts=1671475594.3965611 logger=tls.cache msg=added certificate to cache subjects=["files.wichelo.nl"] expiration=1678543247 managed=true issuer_key=acme-v02.api.letsencrypt.org-directory hash=bfd5eb05fa7ade67d28098b52b560110bfc2ee9c31a3403efd163fb3aa5a0f1a cache_size=1 cache_capacity=10000

DBG ts=1671475594.3966064 logger=events msg=event name=cached_managed_cert id=389e5e32-afb9-478e-a245-727ca64dc3a1 origin=tls data={"sans":["files.wichelo.nl"]}

INF ts=1671475594.396796 msg=autosaved config (load with --resume flag) file=/config/caddy/autosave.json

INF ts=1671475594.3968081 msg=serving initial configuration

Just to make sure I fully understand what is happening:

The ports that I have forwarded are all correct and Caddy uses these to create an outbound request and obtain a certificate
The very same ports (80 and 443) are used by external clients when using my configured DNS (files.wichelo.nl), which should route the request to Caddy
At this point, the certificate that is generated is used to resolve the authenticity of the domain?

My knowledge is quite limited on the last point.

francislavoie · December 19, 2022, 8:25pm

Hmm, really strange.

HTTP seems work fine:

$ curl -v files.wichelo.nl
*   Trying 37.251.33.34:80...
* Connected to files.wichelo.nl (37.251.33.34) port 80 (#0)
> GET / HTTP/1.1
> Host: files.wichelo.nl
> User-Agent: curl/7.81.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://files.wichelo.nl/
< Server: Caddy
< Date: Mon, 19 Dec 2022 20:22:58 GMT
< Content-Length: 0
< 
* Closing connection 0

This correctly shows we’re getting an HTTP->HTTPS redirect from Caddy.

But if we make an HTTPS request, it fails:

$ curl -v https://files.wichelo.nl
*   Trying 37.251.33.34:443...
* Connected to files.wichelo.nl (37.251.33.34) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.0 (OUT), TLS header, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.0 (OUT), TLS header, Unknown (21):
* TLSv1.3 (OUT), TLS alert, decode error (562):
* error:0A000126:SSL routines::unexpected eof while reading
* Closing connection 0
curl: (35) error:0A000126:SSL routines::unexpected eof while reading

Basically, the connection is broken. Curl isn’t receiving any information after sending the initial TLS handshake. This makes me think something in front of your home network is intercepting or cancelling the requests. Maybe your ISP blocks traffic on port 443? You might want to reach out to them and ask.

Hankerdoodle · December 22, 2022, 7:28pm

Interesting. I see your curl response is also different from my own.

This is what I get:

curl -v https://files.wichelo.nl
*   Trying 37.251.33.34:443...
* Connected to files.wichelo.nl (37.251.33.34) port 443 (#0)
* schannel: disabled automatic use of client certificate
* ALPN: offers http/1.1
* schannel: failed to receive handshake, SSL/TLS connection failed
* Closing connection 0
curl: (35) schannel: failed to receive handshake, SSL/TLS connection failed

My ISP shouldn’t block anything, as far as I’ve read. Whether that is actually the case is hard to know, but I will give them a call.

Is it definitely something in front of my home network or could it also be something within? I’d love to exclude at least something to narrow down this search…

francislavoie · December 22, 2022, 8:10pm

Try opening a shell on the server you have Caddy running, then run this:

curl -v --resolve files.wichelo.nl:443:127.0.0.1 https://files.wichelo.nl

This will force the domain to resolve to 127.0.0.1 instead of your WAN IP, and it should reach Caddy directly. If that works, then it’s a problem either with your router or your ISP.

Hankerdoodle · December 23, 2022, 6:21pm

I ran the command a few times. The first two using the ports that are exposed from outside the network (80 and 443)

The second time using the internally exposed ports that have been forwarded to (80 → 8088, and 443 → 8443).

Not sure if the latter makes sense but I can imagine from inside the network these ports should be accessible. The former also puzzles me a bit as the nginx server from the synology apparently forwards the request again to port 5000/5001

emilylange · December 27, 2022, 5:49am

Please try

curl -vL --connect-to files.wichelo.nl:443:127.0.0.1:8443 https://files.wichelo.nl
# and
curl -vL --connect-to files.wichelo.nl:80:127.0.0.1:8080 http://files.wichelo.nl

instead

Hankerdoodle · January 4, 2023, 5:17pm

Sorry for my late reply, had a lot of things on my plate the last few days.
I started completely from scratch and now Caddy is working perfectly!

A few more questions:

I see the following error in the logs when I access the domain. The URL resolves completely fine, but this does worry me slightly:
logger=http msg=looking up info for HTTP challenge host=files.wichelo.nl error=no information found to solve challenge for identifier: files.wichelo.nl)
Accessing the URL by prepending www does not work. How do I fix this?
I have not added the generated certificate to my server. Should I?

francislavoie · January 5, 2023, 5:17am

We’ve seen evidence in the past that ZeroSSL continues making requests for ACME HTTP challenge verification long after it still makes sense to do so. That might be where that log is coming from.

This reminds me that we should probably enhance the logs for that kind of error to show the User-Agent on that request so that we can collect more evidence of ZeroSSL. I just spoke to @matt and we’ll add that in for a future release to we can better keep track of that.

Anyways, ignore this for now, if it still seems like a problem in let’s say a month, feel free to reach out to us again (there’ll probably be a release with the more detailed logging by that point I hope).

That’s a separate domain. You need to configure Caddy to serve that domain as well.

I don’t understand the question. But my guess is “probably not”, but I don’t know what you’re referring to.

matt · January 5, 2023, 5:18am

Done:

@Hankerdoodle Is that your full config? Why do your logs look like… not JSON? Is that logfmt? (Just curious)

Hankerdoodle · January 6, 2023, 7:15pm

@francislavoie thanks for the additional info.
As for the certificate, Synology allows for importing them, which I figured was necessary. I supppose that is all handled by Caddy then?

@matt
My logs are copied directly from Portainer so that definitely formats them itself.

gyfer · January 6, 2023, 11:24pm

You need to manually upload SSL cert & key file into your Synology
which you need to do again when you refresh your SSL cert every few months.

Instead, why not reverse_proxy HTTP/80 at backend (LAN side), and let’s Caddy handle SSL HTTPS ( Front Side, WAN )

if you can access the page http://192.168.2.11:7001 , then

files.wichelo.nl {
	reverse_proxy http://192.168.2.11:7001
}

if you accessing it through insecure https, as https://192.168.2.11:7001 , then

files.wichelo.nl {
	reverse_proxy https://192.168.2.11:7001 {
        transport http {
            tls_insecure_skip_verify
        }
    }
}

Hankerdoodle · January 8, 2023, 4:42pm

Thanks for your answer. I’m still a bit fuzzy on the why, though. So currently the connection is secured, right? What makes it that I would need to manually add it every few months? I currently haven’t added the certificate anywhere within the Synology environment.

Also what is the benefit of not directing to https, through your suggested config? As Caddy already redirects to https that seems way more secure.

Mohammed90 · January 8, 2023, 5:28pm

You don’t need to do anything of that sort. Caddy is managing the certs for you. That interface is for those who want Synology to do the TLS termination with certs obtained from elsewhere. If your system is working, you’re good. Don’t fix something that isn’t broken.

Hankerdoodle · January 9, 2023, 6:32pm

That make sense. Thanks for all the help in this thread. I think we can finally close this topic :).

system · January 12, 2023, 7:32pm

This topic was automatically closed after 30 days. New replies are no longer allowed.