Caddy, LXC, and the PROXY protocol results in TLS handshake errors

Help
1

1. Caddy version (caddy version):

$ caddy version
v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

a. System environment:

I’m running Caddy and the service I’m trying to proxy to (deluge-web) in LXC containers.

  • The host is Ubuntu 20.04
  • The LXC container running Caddy is running Caddy as root in a fresh install of Ubuntu 20.04
  • The LXC container running deluge-web is running as root in a fresh install of Ubuntu 18.04
  • deluge-web is listening on 0.0.0.0:8112 and can be accessed from the LXC container running caddy via it’s interal network
  • A proxy device is listening to 0.0.0.0:80 and 0.0.0.0:443 on the host and forwarding it to the LXC container running caddy

b. Command:

# caddy run

d. My complete Caddyfile or JSON config:

{
        debug
}

my.domain.net, http://192.168.1.204 {
        route /deluge* {
                uri strip_prefix /deluge
                reverse_proxy deluge.lxd:8112 {
                        header_up X-Real-IP {remote}
                        header_up X-Deluge-Base "/deluge"
                }
        }
}

3. The problem I’m having:

This setup works fine, the only problem is that delige-web only see’s the requests coming from 10.140.228.220, which is the IP of the Caddy server. Looking at the DEBUG output form caddy I can see that it’s listing 127.0.0.1 as the remote IP:

2020/08/13 02:45:03.665 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "deluge.lxd:8112", "request": {"method": "GET", "uri": "/", "proto": "HTTP/2.0", "remote_addr": "127.0.0.1:59876", "host": "home.jpolny.net", "headers": {"Sec-Fetch-Site": ["none"], "Sec-Fetch-Mode": ["navigate"], "S
ec-Fetch-User": ["?1"], "X-Forwarded-Proto": ["https"], "Upgrade-Insecure-Requests": ["1"], "Accept-Encoding": ["gzip, deflate, br"], "Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"], "X-Real-Ip": ["127.0.0.1:59876"], "X-Deluge
-Base": ["/deluge"], "Accept-Language": ["en-US,en;q=0.9"], "X-Forwarded-For": ["127.0.0.1"], "User-Agent": ["Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/84.0.4147.108 Safari/537.36"], "Sec-Fetch-Dest": ["document"]}, "tls": {"resumed": false, "version": 772, "ciphersuite": 4865, "p
roto": "h2", "proto_mutual": true, "server_name": "home.jpolny.net"}}, "duration": 0.015309671, "headers": {"Server": ["TwistedWeb"], "Date": ["Thu, 13 Aug 2020 02:45:03 GMT"], "Content-Type": ["text/html; charset=utf-8"], "Content-Length": ["2395"]}, "status": 200}

I tried adding the PROXY protocol to the LXC proxy device, but that results in the following:

# caddy run 
2020/08/13 02:47:37.297 INFO    using adjacent Caddyfile
2020/08/13 02:47:37.298 INFO    admin   admin endpoint started  {"address": "tcp/localhost:2019", "enforce_origin": false, "origins": ["127.0.0.1:2019", "localhost:2019", "[::1]:2019"]}
2020/08/13 02:47:37 [INFO][cache:0xc00058c2a0] Started certificate maintenance routine
2020/08/13 02:47:37.299 INFO    http    server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server {"server_name": "srv1", "http_port": 80}
2020/08/13 02:47:37.299 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2020/08/13 02:47:37.299 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2020/08/13 02:47:37.299 WARN    http    user server is listening on same interface as automatic HTTP->HTTPS redirects; user-configured routes might override these redirects    {"server_name": "srv1", "interface": "tcp/:80"}
2020/08/13 02:47:37.301 INFO    tls     cleaned up storage units
2020/08/13 02:47:37.301 DEBUG   http    starting server loop    {"address": "[::]:443", "http3": false, "tls": true}
2020/08/13 02:47:37.301 DEBUG   http    starting server loop    {"address": "[::]:80", "http3": false, "tls": false}
2020/08/13 02:47:37.301 INFO    http    enabling automatic TLS certificate management   {"domains": ["home.jpolny.net"]}
2020/08/13 02:47:37.315 INFO    autosaved config        {"file": "/root/.config/caddy/autosave.json"}
2020/08/13 02:47:37.315 INFO    serving initial configuration
2020/08/13 02:47:44 http: TLS handshake error from 127.0.0.1:59896: tls: first record does not look like a TLS handshake
2020/08/13 02:47:44 http: TLS handshake error from 127.0.0.1:59898: tls: first record does not look like a TLS handshake
2020/08/13 02:47:45 http: TLS handshake error from 127.0.0.1:59900: tls: first record does not look like a TLS handshake
2020/08/13 02:47:45 http: TLS handshake error from 127.0.0.1:59902: tls: first record does not look like a TLS handshake
2020/08/13 02:47:50 http: TLS handshake error from 127.0.0.1:59904: tls: first record does not look like a TLS handshake
2020/08/13 02:47:50 http: TLS handshake error from 127.0.0.1:59906: tls: first record does not look like a TLS handshake

Based on Proxyprotocol plugin for caddy 2? and the lack of activity on the proxyprotocol module I assume this is something caddy doesn’t directly support, but is there any way to get the real remote IP from the original request?

2

Caddy automatically adds the X-Forwarded-For header to proxied requests, and the backend service should use that to determine the remote IP.

Otherwise, this is something you’ll need to ask the Deluge team to improve, I think.

3

The problem is that Caddy isn’t using the true remote IP in X-Forwarded-For because it’s going through an LXC proxy device.

If I run Caddy directly on the host, I see "X-Forwarded-For": ["WAN IP"] with WAN IP being the IP of the computer I accessed the website from, as expected. When running Caddy in the LXC container, I see "X-Forwarded-For": ["127.0.0.1"]. The same is true for X-Real-IP and remote_addr as well.

This makes sense because the packets are going through a LXC proxy before reaching Caddy. The solution would be using proxy_protocol=true on the LXC proxy device, thus enabling the PROXY protocol and adding X-Forwarded-For before getting to Caddy, but for whatever reason doing this causes TLS handshake errors.

4

Are you able to have your LXC proxy device send the remote IP as some other arbitrary header and then have Caddy handle it with header_up X-Forwarded-For {header.X-Arbitrary-Header} or similar?

I’m not even sure how to debug these. Caddy doesn’t throw that error, Golang’s crypto/tls library does (- The Go Programming Language, CTRL+F “does not look”).

5

I’ve learned today that the PROXY protocol involves adding some bytes before the rest of the HTTPS data, so if a server does not support it, you’ll get TLS handshake errors because it wasn’t able to parse the bytes received.

Currently, adding PROXY protocol support to Caddy would be difficult because Caddy uses the Golang stdlib for TLS, so significant changes would need to be made to capture the bytes for the PROXY protocol before doing the TLS handshake. I’m not saying never though, but probably not soon.

6

Caddy 2 has support for a PROXY protocol module; it would work similar to the v1 plugin did: it wraps the actual net.Listener and when Accept() is called it reads/buffers the first few bytes to handle the PROXY preface, then streams the rest through transparently. Easy to do. Just needs to be done.

7

The plugin @matt is referring to is this one, but it only supports Caddy v1 at the moment. It needs to be ported to Caddy v2.

1 Like
8

This topic was automatically closed after 30 days. New replies are no longer allowed.