Caddy proxy for DoH

Hi,
Trying to use the caddy as a reverse proxy for knot-resolver, DNS over HTTPS.

  1. Caddy version v2.1.1

:
2. How I run Caddy:
caddy run --environ --config /etc/caddy/Caddyfile

The proxy’s relative fragment of Caddyfile:

moscow.acvan.cyou {
	tls /etc/ssl/Letsencrypt/ec/0001_chain.pem /etc/ssl/Letsencrypt/ec/ecc.key
	reverse_proxy /dns-query 127.0.0.1:65533 {
		transport http {
		versions h2c 2
	}
   }
}

The backend Knot-resolver listens HTTPS/2 on 127.0.0.1:65533

Caddyserver uses port 443 for the web-site, so I cannot assign port 443 to Knot-resolver, have to implement proxying by Caddy. The DoH point is https://moscow.acvan.cyou/dns-query

  1. The problem I’m having:
    DNS resolving doesn’t work

  2. Error messages and/or full log output:

feb 05 01:31:34 mail-russia.acvan.net caddy[35545]: {“level”:“error”,“ts”:1612481494.107396,“logger”:“http.log.error”,“msg”:“unexpected EOF”,“request”:{“method”:“POST”,“uri”:"/dns-query",“proto”:“HTTP/2.0”,“remote_addr”:“41.239.9.254:49768”,“host”:“moscow.acvan.cyou”,“headers”:{“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”],“Accept”:[“application/dns-message”],“Accept-Encoding”:[""],“Content-Type”:[“application/dns-message”],“Content-Length”:[“48”]},“tls”:{“resumed”:false,“version”:772,“ciphersuite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.000654693,“status”:502,“err_id”:“7p8bfc9n0”,“err_trace”:“reverseproxy.(*Handler).ServeHTTP (reverseproxy.go:411)”}

You’re using quite an old version, please upgrade to Caddy v2.3.0!

You could probably simplify this to:

reverse_proxy /dns-query* h2c://127.0.0.1:65533

I don’t know much about DoH so I’m not sure I could say much else. I’m not sure if it’s using bog-standard HTTP/2 or if it’s doing some funky stuff causing it to not be proxyable by Caddy (since Caddy just speaks HTTP).

/cc @Mohammed90 who I think may know more, to comment on this

Thank you Francis. I simplified Caddyfile as you advised, and upgraded caddy to 2.3.0. Unfortunately, DoH still not working. Caddy logs somehow changed:

feb 05 02:58:50 mail-russia.acvan.net caddy[37078]: {“level”:“error”,“ts”:1612486730.8215127,“logger”:“http.log.error”,“msg”:“unexpected EOF”,“request”:{“remote_addr”:“41.239.9.254:60836”,“proto”:“HTTP/2.0”,“method”:“POST”,“host”:“moscow.acvan.cyou”,“uri”:"/dns-query",“headers”:{“Accept-Encoding”:[""],“Content-Type”:[“application/dns-message”],“Content-Length”:[“53”],“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”],“Accept”:[“application/dns-message”]},“tls”:{“resumed”:false,“version”:772,“cipher_suite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.001660684,“status”:502,“err_id”:“iqrdc63ih”,“err_trace”:“reverseproxy.statusError (reverseproxy.go:783)”}

OK, I will go on trying to find a solution

I may be wrong, but looking at their docs, it doesn’t look like it supports h2c – I don’t see a way to turn off TLS.

https://knot-resolver.readthedocs.io/en/stable/daemon-bindings-net_tlssrv.html#configuration-options

You might need to use https://127.0.0.1:65533 instead, and configure the TLS options to use the self-signed cert (or tls_insecure_skip_verify I guess, but that’s less ideal)

I’m not sure if that’s the reason for the “unexpected EOF” though, because I’m pretty sure that comes from the Go stdlib (a level deeper than Caddy itself)

Francis is right in that h2c not being support. The IETF RFC 8484 requires the communication to be strictly over https. If you’re using the default of Knot Resolver, then the certificate is self-signed, per their docs:

By default, the web interface starts HTTPS/2 on specified port using an ephemeral TLS certificate that is valid for 90 days and is automatically renewed. It is of course self-signed.

In that case, as Francis said, you’ll need to pass tls_insecure_skip_verify. To validate, I tested Caddy’s capability to proxy DoH to a local instance of CoreDNS with this Caddyfile:

localhost {
	reverse_proxy https://localhost:3000
}

I didn’t need to use the tls_insecure_skip_verify because the certificate used by CoreDNS was already in my system’s trust store. So we know Caddy can handle it, so we just need to troubleshoot the interaction between Caddy and the Knot Resolver.

2 Likes

Thank you very much for explanation. my purpose was to make the Knot to be visible from outside on the standart port 443, unfortunately it seems to be impossible. I will reflect on it, maybe better to proxy Caddy and Knot-resolver via Haproxy or something like that.
Anyway thank you Mohammed and Francis

You could probably do TLS termination?

You mean on Haproxy? I am not sure, it need to be checked

No, with Caddy. I don’t see any reason to introduce haproxy into this at all. Caddy acts as a replacement for haproxy.

Basically what @Mohammed90 was telling you is that you need to run it like this:

Client - HTTPS - Caddy - HTTPS (self-signed) - Knot

What we’re saying is that h2c://, which you were trying, is “HTTP/2 without TLS”, but Knot only allows HTTP/2 with TLS. So you need to use https:// when proxying to it.

1 Like

Ah, that changes the matter, I will try. Lack of my experience, sorry :slight_smile: Thank you, Francis

2 Likes

I failed again :slight_smile: Here is my Caddyfile:

moscow.acvan.cyou {
tls /etc/ssl/Letsencrypt/ec/0000_cert.pem /etc/ssl/Letsencrypt/ec/ecc.key
reverse_proxy /dns-query* https://localhost:65533 {
transport http {
tls_client_auth /etc/knot-resolver/mycert.crt /etc/knot-resolver/mykey.key
}
}
}
mycert.crt and mykey.key are self-signed for domain “localhost”.

When I send DNS queries to the point https://moscow.acvan.cyou/dns-query Caddy says:

feb 06 03:02:42 mail-russia.acvan.net caddy[51245]: {“level”:“error”,“ts”:1612573362.4719424,“logger”:“http.log.error”,“msg”:“x509: certificate is valid for *.acvan.cyou, acvan.cyou, not localhost”,“request”:{“remote_addr”:“194.38.20.218:33182”,“proto”:“HTTP/2.0”,“method”:“POST”,“host”:“moscow.acvan.cyou”,“uri”:"/dns-query",“headers”:{“Accept”:[“application/dns-message”],“Accept-Encoding”:[""],“Content-Type”:[“application/dns-message”],“Content-Length”:[“48”],“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”]},“tls”:{“resumed”:false,“version”:772,“cipher_suite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.002355674,“status”:502,“err_id”:“4q9ta7gt1”,“err_trace”:“reverseproxy.statusError (reverseproxy.go:783)”}

Everything looks correct: Caddy gets a request on the domain with Letsencrypt certificate and proxies it to Knot-resolver, which introduce to Caddy self-signed certificate. Why Caddy mixes different certificates?

You can tidy up your Caddyfile by using the command caddy fmt --config <caddyfile>. It becomes:

moscow.acvan.cyou {
	tls /etc/ssl/Letsencrypt/ec/0000_cert.pem /etc/ssl/Letsencrypt/ec/ecc.key
	reverse_proxy /dns-query* https://localhost:65533 {
		transport http {
			tls_client_auth /etc/knot-resolver/mycert.crt /etc/knot-resolver/mykey.key
		}
	}
}

The directive tls_client_auth is for mTLS authentication. Knot resolver isn’t using mTLS. If you don’t want to use tls_insecure_skip_verify then you should be using tls_trusted_ca_certs /etc/knot-resolver/mycert.crt.

Thank you, I will tidy.
Caddy refuses again. Says the same: “http.log.error”,“msg”:“x509: certificate is valid for *.acvan.cyou,…not for localhost…”. Maybe this is a bug? Because against the logic.
When I changed https://localhost:6533 to 127.0.0.1:65533 it says :“msg”:“x509: cannot validate certificate for 127.0.0.1 because it doesn’t contain any IP SANs”…
OK, I will try " tls_insecure_skip_verify"

Noooo don’t use tls_insecure_skip_verify – that disables security and makes TLS pointless.

Caddy is connecting to your backend https://localhost:65533 and presenting the ServerName of localhost because that is the hostname in the site address. If the backend requires a different server name for some reason, use the tls_server_name option for the http transport: reverse_proxy (Caddyfile directive) — Caddy Documentation

(The backend is presenting a certificate for *.acvan.cyou, acvan.cyou so either you are connecting to the wrong host – and TLS is protecting you – or your client is misconfigured. But don’t don’t don’t disable security!)

Thank you, Matt. Yes, Knot requires domain name, so I made self-signed certificate for moscow.acvan.cyou and changed Caddy like this:

moscow.acvan.cyou {
     tls /etc/ssl/Letsencrypt/ec/0000_cert.pem /etc/ssl/Letsencrypt/ec/ecc.key
            reverse_proxy /dns-query* https://moscow.acvan.cyou:65533 {
                   transport http {
                         tls_trusted_ca_certs /etc/knot-resolver/mycert.crt
                                  }
                  }
}

Caddy, unfortunately, is not happy again :slight_smile: It says:

feb 06 04:21:52 mail-russia.acvan.net caddy[53022]: {“level”:“error”,“ts”:1612578112.30312,“logger”:“http.log.error”,“msg”:“x509: certificate signed by unknown authority”,“request”:{“remote_addr”:“185.86.76.45:55254”,“proto”:“HTTP/2.0”,“method”:“POST”,“host”:“moscow.acvan.cyou”,“uri”:"/dns-query",“headers”:{“Accept”:[“application/dns-message”],“Accept-Encoding”:[""],“Content-Type”:[“application/dns-message”],“Content-Length”:[“48”],“Cache-Control”:[“no-store, no-cache”],“Pragma”:[“no-cache”],“Te”:[“trailers”]},“tls”:{“resumed”:false,“version”:772,“cipher_suite”:4867,“proto”:“h2”,“proto_mutual”:true,“server_name”:“moscow.acvan.cyou”}},“duration”:0.126147698,“status”:502,“err_id”:“v6husheuj”,“err_trace”:“reverseproxy.statusError (reverseproxy.go:783)”}

Still no success, but looks optimistic. Tomorrow will try tls_server_name and other ideas with certificates

That’s because:

So you need to use tls_trusted_ca_certs to tell Caddy it can trust your certificate.

But I used this directive in transport block, didn’t I ? Beyond transport Caddy doesn’t understand it, gets error. Tomorrow again I will try both Letsencrypt certificates or local CA from my Ocserv

You probably configured the wrong cert on either end, then.

It seems that the solution has been found.
The working Caddyfile is:

moscow.acvan.cyou {
	tls /etc/ssl/Letsencrypt/ec/0000_cert.pem /etc/ssl/Letsencrypt/ec/ecc.key
	reverse_proxy /dns-query* https://moscow.acvan.cyou:65533 {
	}
}

The appropriate settings of Knot-resolver are:

net.listen('0.0.0.0', 65533, { kind = 'doh2' })
net.tls("/etc/ssl/Letsencrypt/ec/0000_cert.pem" , "/etc/ssl/Letsencrypt/ec/ecc.key")

DNS queries are being sent to the point https://moscow.acvan.cyou/dns-query

That works, on the server, during querying, netstat indicates this:
> tcp 0 0 62.113.113.34:65533 62.113.113.34:18570 ESTABLISHED 58484/kresd
> tcp6 0 0 62.113.113.34:443 41.239.9.254:56338 ESTABLISHED 58102/caddy

Finally, everything works nice. Thank you for help!

2 Likes

This topic was automatically closed after 30 days. New replies are no longer allowed.