Unbound with DoT and DoH behind Caddy as reverse proxy

qcmr · October 7, 2021, 4:06am

1. Caddy version (`caddy version`):

Caddy v2.4.5

2. How I run Caddy:

I run the official (Alpine) Docker image of Caddy as a reverse proxy to serve web servers or other services on a VPS in the cloud.

a. System environment:

Docker v20.10.9

3. What I want to achieve:

Unbound is a validating, recursive and caching DNS resolver which supports encrypted DNS requests via DNS-over-HTTPS (DoH, port 443) and DNS-over-TLS (DoT, port 853).

My objective is to setup Unbound behind Caddy to resolve downstream encrypted DNS queries coming from my LAN or my mobile phone.

The drawing below illustrates the architecture:

4. My problem:

I am not too sure about the TLS part but it would be very convenient if Caddy could manage the certificate and the key.

Unbound needs to be supplied with the private key for the TLS session (tls-service-key) and the public certificate (tls-service-pem).

5. Solutions?

Feed Unbound with Caddy’s .key and .crt files (of the related domain) located in the Data directory. The only constraint would be to restart Unbound every time the certificates change but a cron job should probably do the job
Leverage local HTTPS (Caddy’s own certificate authority) and feed the certificate to Unbound same as solution 1

Any help would be greatly appreciated so that I understand what I should do regarding this TLS topic.

Thank you!

francislavoie · October 7, 2021, 5:20am

There’s a few tricky bits to this:

Caddy, with the standard installation, ships only an HTTP server. This means it could not terminate DoT because that doesn’t use HTTP. The workaround for this would be to use GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy instead which could allow you to terminate TLS for non-HTTP protocols like DoT. But YMMV, I’ve not heard of anyone trying this yet. Keep in mind caddy-l4 only support JSON configuration at this time, no Caddyfile support yet.
DoH is HTTP as far as I understand, so it should be pretty simple to just use the standard reverse_proxy module to proxy HTTP to the upstream. Caddy can terminate TLS in this case, but I don’t know if Unbound actually requires you to use HTTPS or if it can simply use HTTP, trusting that a proxy is terminating. Either way, you can configure Unbound with a self-signed certificate and just get Caddy to trust it – Unbound doesn’t need to use a publicly trusted cert, because only Caddy would see it anyways.

qcmr · October 7, 2021, 5:37am

DoH is HTTP as far as I understand, so it should be pretty simple to just use the standard reverse_proxy module to proxy HTTP to the upstream.

Yes, that’s right. In my configuration with Unbound on the same VPS as Caddy, it would work, I won´t even need to setup DoH in Unbound. On the other hand, if Unbound is on another VPS and I want to keep the E2E encryption, the self certificate solution is probably the way to go.

The workaround for this would be to use GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy instead which could allow you to terminate TLS for non-HTTP protocols like DoT.

I will dockerize it and give it a try.

Thanks a lot @francislavoie !

Updated diagram with the proposed solution:

francislavoie · October 7, 2021, 5:46am

FYI, if you plan on running 2 instances of Caddy (which is totally valid in this situation I think), then if you share the same storage for both instances of Caddy, then one can initiate certificate issuance and the other can solve it. So that means the “http proxy” instance can complete the issuance for a certificate that the “tcp proxy” instance will need/use.

qcmr · October 7, 2021, 9:18am

Well, actually, it is not exactly that: Unbound does not understand HTTP, only HTTPS (DoH), UDP and TCP. So I don’t have much choice, I have to configure Unbound with a self-signed certificate and get Caddy to trust it.

francislavoie · October 7, 2021, 2:58pm

FYI since it’s only accessible via Caddy, and its inside your own network, you can forgo trust and use the tls_insecure_skip_verify option of reverse_proxy's transport http. Obviously this is less than ideal, but it can save you some effort in making sure Caddy always trusts the self signed cert.

qcmr · October 7, 2021, 3:01pm

Thanks for the tips! I’ll it use if I can’t do it properly… but I am a perfectionist so I will try the hard way

Forza · October 7, 2021, 7:39pm

Please share your configs if you get this to work. It’s quite an interesting idea.

qcmr · October 8, 2021, 2:56am

Yes, will do!

qcmr · October 29, 2021, 3:41am

Hello,

To simplify my tests, a unique Caddy Docker container hosts both the http (for DoH) and tcp (for DoT) proxy for now.

But I am struggling a bit:

The messages I get don’t make sense in relation to the config file:

for the DNS challenge, I specify the staging URL of Let’s Encrypt but Caddy seems to call the prod one.
ZeroSSL warns that providing an email would be nice but I put one already

and more importantly, I can’t get a certificate.

Thanks for your help!

The log messages:

Caddy-l4  | {"level":"info","ts":1635474899.924,"msg":"using provided configuration","config_file":"/etc/caddy/caddy.json","config_adapter":""}
caddy-l4  | {"level":"info","ts":1635474899.9263518,"logger":"admin","msg":"admin endpoint started","address":"tcp/localhost:2019","enforce_origin":false,"origins":["localhost:2019","[::1]:2019","127.0.0.1:2019"]}
caddy-l4  | {"level":"info","ts":1635474899.9277084,"logger":"http","msg":"enabling automatic HTTP->HTTPS redirects","server_name":"unbound-doh"}
caddy-l4  | {"level":"info","ts":1635474899.9294147,"logger":"http","msg":"enabling automatic TLS certificate management","domains":["https://dns.mydomaine.com"]}
caddy-l4  | {"level":"info","ts":1635474899.9303,"msg":"autosaved config (load with --resume flag)","file":"/config/caddy/autosave.json"}
caddy-l4  | {"level":"info","ts":1635474899.9306996,"msg":"serving initial configuration"}
caddy-l4  | {"level":"info","ts":1635474899.9314098,"logger":"tls.cache.maintenance","msg":"started background certificate maintenance","cache":"0xc00027ea10"}
caddy-l4  | {"level":"info","ts":1635474899.9317787,"logger":"tls","msg":"cleaning storage unit","description":"FileStorage:/data/caddy"}
caddy-l4  | {"level":"info","ts":1635474899.9322598,"logger":"tls","msg":"finished cleaning storage units"}
caddy-l4  | {"level":"info","ts":1635474899.9332206,"logger":"tls.obtain","msg":"acquiring lock","identifier":"https://dns.mydomaine.com"}
caddy-l4  | {"level":"info","ts":1635474899.9418845,"logger":"tls.obtain","msg":"lock acquired","identifier":"https://dns.mydomaine.com"}
caddy-l4  | {"level":"info","ts":1635474900.6414804,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["https://dns.mydomaine.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy-l4  | {"level":"info","ts":1635474900.6419895,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["https://dns.mydomaine.com"],"ca":"https://acme-v02.api.letsencrypt.org/directory","account":""}
caddy-l4  | {"level":"error","ts":1635474900.6423519,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"https://dns.mydomaine.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[https://dns.mydomaine.com] no identifiers found (ca=https://acme-v02.api.letsencrypt.org/directory)"}
caddy-l4  | {"level":"warn","ts":1635474900.6428435,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
caddy-l4  | {"level":"info","ts":1635474901.6937494,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"TNMr5rHBPVTFAisiMsjL0A"}
caddy-l4  | {"level":"info","ts":1635474903.0573525,"logger":"tls.issuance.acme","msg":"waiting on internal rate limiter","identifiers":["https://dns.mydomaine.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
caddy-l4  | {"level":"info","ts":1635474903.05879,"logger":"tls.issuance.acme","msg":"done waiting on internal rate limiter","identifiers":["https://dns.mydomaine.com"],"ca":"https://acme.zerossl.com/v2/DV90","account":""}
caddy-l4  | {"level":"error","ts":1635474903.0591238,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"https://dns.mydomaine.com","issuer":"acme.zerossl.com-v2-DV90","error":"[https://dns.mydomaine.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)"}
caddy-l4  | {"level":"error","ts":1635474903.0595586,"logger":"tls.obtain","msg":"will retry","error":"[https://dns.mydomaine.com] Obtain: [https://dns.mydomaine.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)","attempt":1,"retrying_in":60,"elapsed":3.117370822,"max_duration":2592000}
caddy-l4  | {"level":"error","ts":1635474963.7021089,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"https://dns.mydomaine.com","issuer":"acme-v02.api.letsencrypt.org-directory","error":"[https://dns.mydomaine.com] no identifiers found (ca=https://acme-staging-v02.api.letsencrypt.org/directory)"}
caddy-l4  | {"level":"warn","ts":1635474963.7033336,"logger":"tls.issuance.zerossl","msg":"missing email address for ZeroSSL; it is strongly recommended to set one for next time"}
caddy-l4  | {"level":"info","ts":1635474964.696057,"logger":"tls.issuance.zerossl","msg":"generated EAB credentials","key_id":"l3-vLBnh_Uhy4f9heCoOWw"}
caddy-l4  | {"level":"error","ts":1635474965.7438407,"logger":"tls.obtain","msg":"could not get certificate from issuer","identifier":"https://dns.mydomaine.com","issuer":"acme.zerossl.com-v2-DV90","error":"[https://dns.mydomaine.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)"}
caddy-l4  | {"level":"error","ts":1635474965.7446706,"logger":"tls.obtain","msg":"will retry","error":"[https://dns.mydomaine.com] Obtain: [https://dns.mydomaine.com] no identifiers found (ca=https://acme.zerossl.com/v2/DV90)","attempt":2,"retrying_in":120,"elapsed":65.802481174,"max_duration":2592000}

Caddy config file:

{
	"apps": {
		"tls": {
			"automation": {
				"policies": [
					{
						"subjects": [
							"dns.mydomaine.com"
						],
						"issuers": [
							{
								"module": "acme",
								"ca": "https://acme-staging-v02.api.letsencrypt.org/directory",
								"test_ca": "https://acme-staging-v02.api.letsencrypt.org/directory",
								"email": "contact@myemail.com",
								"acme_timeout": 0,
								"challenges": {
									"http": {
										"disabled": true,
										"alternate_port": 0
									},
									"tls-alpn": {
										"disabled": true,
										"alternate_port": 0
									},
									"dns": {
										"provider": {
											"name": "gandi",
											"api_token": "{env.GANDI_API_TOKEN}"
										},
										"ttl": 0,
										"propagation_timeout": 0
									}
								}
							},
							{
								"module": "zerossl",
								"ca": "https://acme.zerossl.com/v2/DV90",
								"test_ca": "",
								"email": "contact@myemail.com",
								"acme_timeout": 0,
								"challenges": {
									"http": {
										"disabled": true,
										"alternate_port": 0
									},
									"tls-alpn": {
										"disabled": true,
										"alternate_port": 0
									},
									"dns": {
										"provider": {
											"name": "gandi",
											"api_token": "{env.GANDI_API_TOKEN}"
										},
										"ttl": 0,
										"propagation_timeout": 0
									}
								}
							}
						]
					}
				]
			}
		},
		"http": {
			"http_port": 80,
			"https_port": 443,
			"servers": {
				"unbound-doh": {
					"listen": [
						":443"
					],
					"routes": [
						{
							"match": [
								{
									"host": [
										"https://dns.mydomaine.com"
									]
								}
							],
							"handle": [
								{
									"handler": "rewrite",
									"uri": "/dns-query"
								},
								{
									"handler": "reverse_proxy",
									"transport": {
										"protocol": "http",
										"compression": true,
										"versions": [
											"2"
										]
									},
									"upstreams": [
										{
											"dial": "localhost:8443"
										}
									]
								}
							],
							"terminal": false
						}
					],
					"tls_connection_policies": [
						{
							"protocol_min": "tls1.3"
						}
					]
				}
			}
		},
		"layer4": {
			"servers": {
				"unbound-dot": {
					"listen": [
						":853"
					],
					"routes": [
						{
							"match": [
								{
									"tls": {
										"sni": [
											"dns.mydomaine.com"
										]
									}
								}
							],
							"handle": [
								{
									"handler": "proxy",
									"upstreams": [
										{
											"dial": [
												"localhost:8853"
											]
										}
									]
								}
							]
						}
					]
				}
			}
		}
	}
}

Dockerfile:

FROM caddy:2.4.5-builder-alpine AS builder

RUN xcaddy build \
    --with github.com/mholt/caddy-l4 \
    --with github.com/caddy-dns/gandi

FROM caddy:2.4.5-alpine

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

COPY caddy.json /etc/caddy/caddy.json

CMD ["caddy", "run", "--config", "/etc/caddy/caddy.json"]

docker-compose.yml:

version: "3.9"

services:
    caddy-l4:
        build: .
        container_name: caddy-l4
        hostname: caddy-l4
        ports:
            - "80:80"
            - "443:443"
            - "853:853"
        environment:
            - GANDIV5_API_KEY
        volumes:
            - ./caddy-l4/data:/data
            - ./caddy-l4/config:/config
            - ./caddy-l4/caddy.json:/etc/caddy/caddy.json
        networks:
            - proxy
networks:
    proxy:

francislavoie · October 29, 2021, 5:03am

Your host matcher should only include the domain, and not the scheme.

The no identifiers found error is because that couldn’t be parsed as a valid domain (because it also has the scheme which is extra).

qcmr · October 29, 2021, 5:47am

@francislavoie : Thanks a lot!

system · November 6, 2021, 4:06am

This topic was automatically closed after 30 days. New replies are no longer allowed.