Turning off Cloudflare ECH fixed some errors!

[GeraldB (gboss)]

1. The problem I’m having:

I have a similar/the same (?) problem as mentioned here: Ssl_error_interal_error_alert
@ panoskpv
I host open source apps in my home-lab in a quite complex setup and a few weeks ago my browsers sometimes returned SSL Errors …
A short overview of my setup:

• internet access with dynamic ip, freedns provides a cname I use with duckdns to get acme dns challenge for my domain that I host at cloudflare…
• rhel-server with podman (private ip in dmz of my firewall)
• caddy running in a container (2.9.3 beta built with duckdns-plugin) as reverse proxy for…
• apps, running in pods and containers
• cloudflared with zerotrus applications pointing to localhost:[app-port] of my rhel-server to secure access to my apps from the internet
• local access from LAN through my firewall which also is the local DNS, sending local requests for the official hostnames of my apps to the local rhel-server

Some weeks ago, when I tried to access my apps (e.g. FreshRSS) I got SSL Errors. But. Not always. It took me some nights to find out, that it was some sort of handshake error concerning cloudflares encrypted client hello (ECH)…
Btw with curl or openssl the problem did not exist > caddy always returned the html-code form the app. But at the same time every browser returned an SSL error message…

Some nights later I found an article online that cloudflare turned on ECH in the “free plan” and the only way to turn it off for my domain/zone was to use the cloudflare API.
After turning ECH off, SSL handshake with my browsers worked again (not only sometimes) and my apps are accessible again … no error for 24 hours, yay!

2. Error messages and/or full log output:

{"level":"debug","ts":1731759643.5468824,"logger":"events","msg":"event","name":"tls_get_certificate","id":"----id----","origin":"tls","data":{"client_hello":{"CipherSuites":[56026,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"ServerName":"cloudflare-ech.com","SupportedCurves":[64250,29,23,24],"SupportedPoints":"AA==","SignatureSchemes":[1027,2052,1025,1283,2053,1281,2054,1537],"SupportedProtos":null,"SupportedVersions":[771,770,769],"RemoteAddr":{"IP":"192.168.1.37","Port":49532,"Zone":""},"LocalAddr":{"IP":"10.0.2.100","Port":443,"Zone":""}}}}
{"level":"debug","ts":1731759643.547051,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"cloudflare-ech.com"}
{"level":"debug","ts":1731759643.547081,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.com"}
{"level":"debug","ts":1731759643.5470986,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"*.*"}
{"level":"debug","ts":1731759643.5471168,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","remote_ip":"192.168.1.37","remote_port":"49532","server_name":"cloudflare-ech.com","remote":"192.168.1.37:49532","identifier":"cloudflare-ech.com","cipher_suites":[56026,4865,4866,4867,49195,49199,49196,49200,52393,52392,49171,49172,156,157,47,53],"cert_cache_fill":0.002,"load_or_obtain_if_necessary":true,"on_demand":false}

3. Caddy version:

v2.9.0-beta.3.0.20241115034942-6028ff27fa8e h1:JMuHFiHz3F6ldr0nsRS92GRSzcsBEWdgXtxxQjRgbhc=

4. How I installed and ran Caddy:

podman, custom build …

RUN xcaddy build master \
    --with github.com/caddy-dns/duckdns \
    --with github.com/caddy-dns/cloudflare \
    --with github.com/quic-go/quic-go=github.com/WeidiDeng/quic-go@shutdown-fix

FROM docker.io/caddy:latest

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

a. System environment:

RHEL 8.10
podman 4.9.4

b. Command:

c. Service/unit/compose file:

d. My complete Caddy config: (no, it is not complete, only one app )

# global options
{
        # debug
        acme_dns duckdns [mysecretkey] {
                override_domain [secretcname].duckdns.org
        }
        email myemail@mydomain
        servers {
                client_ip_headers X-Forwarded-For X-Real-IP
                log_credentials
        }
}
#ttrss
ttrss.mydomain.com {
        reverse_proxy 192.168.2.1:8089
}

5. Links to relevant resources:

Support Encrypted Client Hello (formerly known as ESNI)

Thanks for the API Path/Attribute/Value that I did not find in the official cloudflare api docs …:

Disabling Encrypted Client Hello (ECH) for Cloudflare Domains