Proxying HTTPS traffic to another caddy instance/split DNS

1. Caddy version (caddy version):

2.4.5

2. How I run Caddy:

Installed via official Ansible role

a. System environment:

Ubuntu 20.04 LTS.

b. Command:

/usr/local/bin/caddy" run --environ --config "/home/fuzzy/caddy/Caddyfile

c. Service/unit/compose file:

;
; Ansible managed
;
; source: https://github.com/mholt/caddy/blob/master/dist/init/linux-systemd/caddy.service
; version: 6be0386
; changes: Set variables via Ansible

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network-online.target

[Service]
Restart=on-failure
StartLimitInterval=86400
StartLimitBurst=5

; User and group the process will run as.
User=www-data
Group=www-data

; Letsencrypt-issued certificates will be written to this directory.
Environment=CADDYPATH=/etc/ssl/caddy

ExecStart="/usr/local/bin/caddy" run --environ --config "/home/fuzzy/caddy/Caddyfile"
ExecReload="/usr/local/bin/caddy" reload --config "/home/fuzzy/caddy/Caddyfile"

; Limit the number of file descriptors; see `man systemd.exec` for more limit settings.
LimitNOFILE=1048576

; Use private /tmp and /var/tmp, which are discarded after caddy stops.
PrivateTmp=true
; Use a minimal /dev
PrivateDevices=true
; Hide /home, /root, and /run/user. Nobody will steal your SSH-keys.
ProtectHome=false
; Make /usr, /boot, /etc and possibly some more folders read-only.
ProtectSystem=full
;    ^`    except /etc/ssl/caddy, because we want Letsencrypt-certificates there.
;   This merely retains r/w access rights, it does not add any new. Must still be writable on the host!
ReadWriteDirectories=/etc/ssl/caddy /var/log/caddy

; The following additional security directives only work with systemd v229 or later.
; They further retrict privileges that can be gained by caddy.
; Note that you may have to add capabilities required by any plugins in use.
CapabilityBoundingSet=CAP_NET_BIND_SERVICE
AmbientCapabilities=CAP_NET_BIND_SERVICE
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

d. My complete Caddyfile or JSON config:

Links provided below if needed

3. The problem I’m having:

I run Caddy in two places, on my VPS which is public facing and then on a local server that has a wireguard connection to the VPS. What I’d like to do is be able to set up my Adguard/DNS on my home network to point to the local server so I avoid putting traffic through the VPS (and also eliminate as much latency as possible). Currently I have a setup that’s working but it terminates TLS on the VPS and forwards the traffic to the local Caddy server over http. I’d like to also have HTTPS when i’m on my home network.

What I’d ideally like to do is have only reverse_proxy entry per domain. IE just something like this:

photos.example.com {
        tls PATH_TO_CERTS
        reverse_proxy http://192.168.1.20:8000
        import headers
}

That could handle both the connection from the VPS and internally (and give me HTTPS in both spots). I figured out a solution using request matching with something like this:

https://photos.example.com {
        @local remote_ip 192.168.0.0/16 172.16.0.0/12 10.0.0.0/8
        reverse_proxy @local http://192.168.1.20:8000
        tls PATH_TO_CERTS
        import headers
}
http://photos.example.com {
        reverse_proxy http://192.168.1.20:8000
        import headers
}

But that’s kludgey on a large scale. Is there a way to streamline this to be a single block per subdomain? Or alternatively, is it possible to do HTTPS on both the VPS and local server so it wouldn’t matter? I tried something like this on the VPS/local server to do HTTPS/HTTPS but all I got was a blank page:

VPS:

*.example.com {
        tls PATH_TO_CERTS

        @photos host photos.example.com
        handle @photos {
                reverse_proxy https://10.10.10.10:443
                import personal_headers
                import no_robots
        }
}

Local server:

photos.example.com {
        tls PATH_TO_CERTS
        reverse_proxy http://192.168.1.20:8000
        import headers
}

If you really want to see my full Caddyfiles of my existing setup (that’s HTTPS->HTTP), here’s the VPS one and the local server one

4. Error messages and/or full log output:

No errors that I can see

5. What I already tried:

See above

6. Links to relevant resources:

If you got a blank page, it’s probably because some matcher didn’t match, and the request fell through and was served with an empty response (Caddy’s default – because it essentially did what it was configured to do, i.e. nothing).

Turn on the debug global option on both instances and see what appears in the logs.

This seems to be the issue from the VPS log:

{"level":"error","ts":1633088221.9123425,"logger":"http.log.access.log0","msg":"handled request","request":{"remote_addr":"IP_ADDRESS:27622","proto":"HTTP/2.0","method":"GET","host":"photos.example.com","uri":"/","headers":{"Sec-Fetch-Dest":["document"],"Cache-Control":["max-age=0"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Sec-Fetch-User":["?1"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Dnt":["1"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"photos.example.com"}},"common_log":"IP_ADDRESS - - [01/Oct/2021:07:37:01 -0400] \"GET / HTTP/2.0\" 502 0","user_id":"","duration":0.021759733,"size":0,"status":502,"resp_headers":{"Server":["Caddy"]}}

Nothing on the local server log.

Did you turn debug logging? Add this at the top of both of your Caddyfiles:

{
	debug
}

It should make it clearer why a 502 is being returned.

Only thing I can see in the logs is this:

On the VPS:

{"level":"debug","ts":1633139766.9580257,"logger":"tls.handshake","msg":"no matching certificate; will choose from all certificates","identifier":"photos.example.com"}
{"level":"debug","ts":1633139766.9580798,"logger":"tls.handshake","msg":"choosing certificate","identifier":"photos.example.com","num_choices":3}
{"level":"debug","ts":1633139766.9580991,"logger":"tls.handshake","msg":"custom certificate selection results","identifier":"photos.example.com","subjects":["*.example.com"],"managed":false,"issuer_key":"","hash":"9380d431733726e4d3ebcabfc2f95ad3ee71c56ffd377f9818ff6952997183f7"}
{"level":"debug","ts":1633139766.9581094,"logger":"tls.handshake","msg":"matched certificate in cache","subjects":["*.example.com"],"managed":false,"expiration":1640902065,"hash":"9380d431733726e4d3ebcabfc2f95ad3ee71c56ffd377f9818ff6952997183f7"}
{"level":"debug","ts":1633139766.9958613,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.10.10:443","request":{"remote_addr":"IP_ADDRESS:5208","proto":"HTTP/2.0","method":"GET","host":"photos.example.com","uri":"/","headers":{"Sec-Fetch-User":["?1"],"Te":["trailers"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Sec-Fetch-Mode":["navigate"],"X-Forwarded-For":["IP_ADDRESS"],"X-Forwarded-Proto":["https"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Site":["none"],"Cache-Control":["max-age=0"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"photos.example.com"}},"duration":0.020815065,"error":"remote error: tls: internal error"}
{"level":"error","ts":1633139766.9960778,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_addr":"IP_ADDRESS:5208","proto":"HTTP/2.0","method":"GET","host":"photos.example.com","uri":"/","headers":{"Sec-Fetch-Site":["none"],"Cache-Control":["max-age=0"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"Sec-Fetch-Mode":["navigate"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Sec-Fetch-User":["?1"],"Te":["trailers"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"photos.example.com"}},"duration":0.021214735,"status":502,"err_id":"eiwm2p68r","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}

On the local server:

{"level":"debug","ts":1633139707.152601,"logger":"tls.handshake","msg":"no matching certificates and no custom selection logic","identifier":"10.10.10.10"}
{"level":"debug","ts":1633139707.152627,"logger":"tls.handshake","msg":"no certificate matching TLS ClientHello","server_name":"","remote":"10.10.10.1:49116","identifier":"10.10.10.10","cipher_suites":[49195,49199,49196,49200,52393,52392,49161,49171,49162,49172,156,157,47,53,49170,10,4865,4866,4867],"cache_size":1,"cache_capacity":10000,"load_if_necessary":true,"obtain_if_necessary":true,"on_demand":false}
{"level":"debug","ts":1633139707.1526608,"logger":"http.stdlib","msg":"http: TLS handshake error from 10.10.10.1:49116: no certificate available for '10.10.10.10'"}

Aaah right, I think you’ll need to explicitly set the Host header so that TLS SNI uses the right hostname.

reverse_proxy https://10.10.10.10:443 {
    header_up Host {hostport}
}

Added to the VPS Caddyfile:

        @photos host photos.example.com
        handle @photos {
                reverse_proxy https://10.10.10.10:443 {
                        header_up Host {hostport}
                }
                import personal_headers
                import no_robots
        }

Get this:

{"level":"debug","ts":1633142241.547122,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.10.10:443","request":{"remote_addr":"IP_ADDRESS:6954","proto":"HTTP/2.0","method":"GET","host":"photos.example.com","uri":"/","headers":{"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"X-Forwarded-Proto":["https"],"Sec-Fetch-User":["?1"],"Accept-Encoding":["gzip, deflate, br"],"Dnt":["1"],"X-Forwarded-For":["IP_ADDRESS"],"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-Site":["none"],"Accept-Language":["en-US,en;q=0.5"],"Sec-Fetch-Dest":["document"],"Te":["trailers"],"Upgrade-Insecure-Requests":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"photos.example.com"}},"duration":0.018884037,"error":"remote error: tls: internal error"}
{"level":"error","ts":1633142241.5472481,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_addr":"IP_ADDRESS:6954","proto":"HTTP/2.0","method":"GET","host":"photos.example.com","uri":"/","headers":{"Sec-Fetch-Mode":["navigate"],"Sec-Fetch-User":["?1"],"Accept-Language":["en-US,en;q=0.5"],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-Dest":["document"],"Dnt":["1"],"Sec-Fetch-Site":["none"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"],"Accept-Encoding":["gzip, deflate, br"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"photos.example.com"}},"duration":0.019211007,"status":502,"err_id":"x461363e8","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}
{"level":"debug","ts":1633142241.6082606,"logger":"http.handlers.reverse_proxy","msg":"upstream roundtrip","upstream":"10.10.10.10:443","request":{"remote_addr":"IP_ADDRESS:6954","proto":"HTTP/2.0","method":"GET","host":"photos.example.com","uri":"/favicon.ico","headers":{"Te":["trailers"],"Accept-Encoding":["gzip, deflate, br"],"Sec-Fetch-Dest":["image"],"X-Forwarded-Proto":["https"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"],"Dnt":["1"],"X-Forwarded-For":["IP_ADDRESS"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Accept":["image/webp,*/*"],"Accept-Language":["en-US,en;q=0.5"],"Referer":["https://photos.example.com/"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"photos.example.com"}},"duration":0.020317127,"error":"remote error: tls: internal error"}
{"level":"error","ts":1633142241.6084,"logger":"http.log.error","msg":"remote error: tls: internal error","request":{"remote_addr":"IP_ADDRESS:6954","proto":"HTTP/2.0","method":"GET","host":"photos.example.com","uri":"/favicon.ico","headers":{"Accept":["image/webp,*/*"],"Accept-Language":["en-US,en;q=0.5"],"Accept-Encoding":["gzip, deflate, br"],"Referer":["https://photos.example.com/"],"Sec-Fetch-Dest":["image"],"Sec-Fetch-Mode":["no-cors"],"Sec-Fetch-Site":["same-origin"],"Te":["trailers"],"User-Agent":["Mozilla/5.0 (X11; Linux x86_64; rv:92.0) Gecko/20100101 Firefox/92.0"],"Dnt":["1"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","proto_mutual":true,"server_name":"photos.example.com"}},"duration":0.020591997,"status":502,"err_id":"p0hcmxynh","err_trace":"reverseproxy.statusError (reverseproxy.go:858)"}

Hmm. I guess it would be this instead:

reverse_proxy https://10.10.10.10:443 {
	transport http {
		tls_server_name photos.example.com
	}
}

Unfortunately, placeholders can’t be used for tls_server_name, because the TLS config needs to be provisioned on config load rather than on every request.

That did it, thank you! It isn’t “clean” but that’s simpler than my other approach.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.