The backend Knot-resolver listens HTTPS/2 on 127.0.0.1:65533
Caddyserver uses port 443 for the web-site, so I cannot assign port 443 to Knot-resolver, have to implement proxying by Caddy. The DoH point is https://moscow.acvan.cyou/dns-query
The problem I’m having:
DNS resolving doesn’t work
Error messages and/or full log output:
feb 05 01:31:34 mail-russia.acvan.net caddy[35545]: {“level”:“error”,“ts”:1612481494.107396,“logger”:“http.log.error”,“msg”:“unexpected EOF”,“request”:{“method”:“POST”,“uri”:“/dns-query”,“proto”:“HTTP/2.0”,“remote_addr”:“41.239.9.254:49768”,“host”:“moscow.acvan.cyou”,“headers”:{“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”],“Accept”:[“application/dns-message”],“Accept-Encoding”:[“”],“Content-Type”:[“application/dns-message”],“Content-Length”:[“48”]},“tls”:{“resumed”:false,“version”:772,“ciphersuite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.000654693,“status”:502,“err_id”:“7p8bfc9n0”,“err_trace”:“reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:411)”}
You’re using quite an old version, please upgrade to Caddy v2.3.0!
You could probably simplify this to:
reverse_proxy /dns-query* h2c://127.0.0.1:65533
I don’t know much about DoH so I’m not sure I could say much else. I’m not sure if it’s using bog-standard HTTP/2 or if it’s doing some funky stuff causing it to not be proxyable by Caddy (since Caddy just speaks HTTP).
/cc @Mohammed90 who I think may know more, to comment on this
Thank you Francis. I simplified Caddyfile as you advised, and upgraded caddy to 2.3.0. Unfortunately, DoH still not working. Caddy logs somehow changed:
feb 05 02:58:50 mail-russia.acvan.net caddy[37078]: {“level”:“error”,“ts”:1612486730.8215127,“logger”:“http.log.error”,“msg”:“unexpected EOF”,“request”:{“remote_addr”:“41.239.9.254:60836”,“proto”:“HTTP/2.0”,“method”:“POST”,“host”:“moscow.acvan.cyou”,“uri”:“/dns-query”,“headers”:{“Accept-Encoding”:[“”],“Content-Type”:[“application/dns-message”],“Content-Length”:[“53”],“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”],“Accept”:[“application/dns-message”]},“tls”:{“resumed”:false,“version”:772,“cipher_suite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.001660684,“status”:502,“err_id”:“iqrdc63ih”,“err_trace”:“reverseproxy.statusError (reverseproxy.go:783)”}
You might need to use https://127.0.0.1:65533 instead, and configure the TLS options to use the self-signed cert (or tls_insecure_skip_verify I guess, but that’s less ideal)
I’m not sure if that’s the reason for the “unexpected EOF” though, because I’m pretty sure that comes from the Go stdlib (a level deeper than Caddy itself)
Francis is right in that h2c not being support. The IETF RFC 8484 requires the communication to be strictly over https. If you’re using the default of Knot Resolver, then the certificate is self-signed, per their docs:
By default, the web interface starts HTTPS/2 on specified port using an ephemeral TLS certificate that is valid for 90 days and is automatically renewed. It is of course self-signed.
In that case, as Francis said, you’ll need to pass tls_insecure_skip_verify. To validate, I tested Caddy’s capability to proxy DoH to a local instance of CoreDNS with this Caddyfile:
I didn’t need to use the tls_insecure_skip_verify because the certificate used by CoreDNS was already in my system’s trust store. So we know Caddy can handle it, so we just need to troubleshoot the interaction between Caddy and the Knot Resolver.
Thank you very much for explanation. my purpose was to make the Knot to be visible from outside on the standart port 443, unfortunately it seems to be impossible. I will reflect on it, maybe better to proxy Caddy and Knot-resolver via Haproxy or something like that.
Anyway thank you Mohammed and Francis
What we’re saying is that h2c://, which you were trying, is “HTTP/2 without TLS”, but Knot only allows HTTP/2 with TLS. So you need to use https:// when proxying to it.
feb 06 03:02:42 mail-russia.acvan.net caddy[51245]: {“level”:“error”,“ts”:1612573362.4719424,“logger”:“http.log.error”,“msg”:“x509: certificate is valid for *.acvan.cyou, acvan.cyou, not localhost”,“request”:{“remote_addr”:“194.38.20.218:33182”,“proto”:“HTTP/2.0”,“method”:“POST”,“host”:“moscow.acvan.cyou”,“uri”:“/dns-query”,“headers”:{“Accept”:[“application/dns-message”],“Accept-Encoding”:[“”],“Content-Type”:[“application/dns-message”],“Content-Length”:[“48”],“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”]},“tls”:{“resumed”:false,“version”:772,“cipher_suite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.002355674,“status”:502,“err_id”:“4q9ta7gt1”,“err_trace”:“reverseproxy.statusError (reverseproxy.go:783)”}
Everything looks correct: Caddy gets a request on the domain with Letsencrypt certificate and proxies it to Knot-resolver, which introduce to Caddy self-signed certificate. Why Caddy mixes different certificates?
The directive tls_client_auth is for mTLS authentication. Knot resolver isn’t using mTLS. If you don’t want to use tls_insecure_skip_verify then you should be using tls_trusted_ca_certs /etc/knot-resolver/mycert.crt.
Thank you, I will tidy.
Caddy refuses again. Says the same: “http.log.error”,“msg”:“x509: certificate is valid for *.acvan.cyou,…not for localhost…”. Maybe this is a bug? Because against the logic.
When I changed https://localhost:6533 to 127.0.0.1:65533 it says :“msg”:“x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs”…
OK, I will try " tls_insecure_skip_verify"
Noooo don’t use tls_insecure_skip_verify – that disables security and makes TLS pointless.
Caddy is connecting to your backend https://localhost:65533 and presenting the ServerName of localhost because that is the hostname in the site address. If the backend requires a different server name for some reason, use the tls_server_name option for the http transport: reverse_proxy (Caddyfile directive) — Caddy Documentation
(The backend is presenting a certificate for *.acvan.cyou, acvan.cyou so either you are connecting to the wrong host – and TLS is protecting you – or your client is misconfigured. But don’t don’t don’t disable security!)
feb 06 04:21:52 mail-russia.acvan.net caddy[53022]: {“level”:“error”,“ts”:1612578112.30312,“logger”:“http.log.error”,“msg”:“x509: certificate signed by unknown authority”,“request”:{“remote_addr”:“185.86.76.45:55254”,“proto”:“HTTP/2.0”,“method”:“POST”,“host”:“moscow.acvan.cyou”,“uri”:“/dns-query”,“headers”:{“Accept”:[“application/dns-message”],“Accept-Encoding”:[“”],“Content-Type”:[“application/dns-message”],“Content-Length”:[“48”],“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”]},“tls”:{“resumed”:false,“version”:772,“cipher_suite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.126147698,“status”:502,“err_id”:“v6husheuj”,“err_trace”:“reverseproxy.statusError (reverseproxy.go:783)”}
Still no success, but looks optimistic. Tomorrow will try tls_server_name and other ideas with certificates
But I used this directive in transport block, didn’t I ? Beyond transport Caddy doesn’t understand it, gets error. Tomorrow again I will try both Letsencrypt certificates or local CA from my Ocserv
DNS queries are being sent to the point https://moscow.acvan.cyou/dns-query
That works, on the server, during querying, netstat indicates this:
> tcp 0 0 62.113.113.34:65533 62.113.113.34:18570 ESTABLISHED 58484/kresd
> tcp6 0 0 62.113.113.34:443 41.239.9.254:56338 ESTABLISHED 58102/caddy
Finally, everything works nice. Thank you for help!
Thank you, Francis
Here is my Caddyfile:
It says: