Caddy to another caddy

Help
1

Is it possible to reverse proxy to another caddy instance which in turn reverse proxies the request?

1. My Caddy version (caddy version):

V2 from the latest dockerhub image

2. How I run Caddy:

A caddy on a droplet configured to reverse to another caddy as follows:

    https://gateway.host {
      log {
        output stdout
        level debug
      }
      reverse_proxy {
        to https://service.host:28989 
      }
    }

Caddy on another server configured as follows:

    https://service.host:28989 {
      log {
        output file /data/logs/service.log
        level debug
      }
      reverse_proxy {
        to http://localhost:1234
       }
    }

a. System environment:

both servers are ubuntu linux dists, systemd and docker with the same docker compose.

c. Service/unit/compose file:

caddy uses host networking and starts up correctly. Other configurations of caddy work flawlessly.

3. The problem I’m having:

Reach the service running on service.host:1234 through gateway.host.

A browser returns a blank page with what looks like a status 200. It does show a curious message in the console which isn’t normally there:
The character encoding of the plain text document was not declared. The document will render with garbled text in some browser configurations if the document contains characters from outside the US-ASCII range. The character encoding of the file needs to be declared in the transfer protocol or file needs to use a byte order mark as an encoding signature.

Curl simply returns what looks like a message from Caddy:

* found 149 certificates in /etc/ssl/certs/ca-certificates.crt
* found 597 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* 	 common name: service.host (matched)
* 	 server certificate expiration date OK
* 	 server certificate activation date OK
* 	 certificate public key: EC
* 	 certificate version: #3
* 	 subject: CN=service.host
* 	 start date: Sun, 29 Mar 2020 13:10:09 GMT
* 	 expire date: Sat, 27 Jun 2020 13:10:09 GMT
* 	 issuer: C=US,O=Let's Encrypt,CN=Let's Encrypt Authority X3
* 	 compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host: sonarr.mjd.one
> User-Agent: curl/7.47.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Content-Length: 0
< Date: Sun, 29 Mar 2020 17:28:25 GMT
< Server: Caddy
< Server: Caddy

If you are wondering why I’ve got two caddy instances it’s because the gateway accepts ipv4 (gateway.host) and the service.host only accepts ipv6. It’s essentially an ipv4 to ipv6 network bridge.

4. Error messages and/or full log output:

From the gateway caddy:

 gateway-caddy | {
        "level": "info",
        "ts": 1585501669.1829925,
        "logger": "http.log.access.log1",
        "msg": "handled request",
        "request": {
            "method": "GET",
            "uri": "/",
            "proto": "HTTP/2.0",
            "remote_addr": "xxxxxxxx:xxxx",
            "host": "gateway.host",
            "headers": {
                "Te": [
                    "trailers"
                ],
                "User-Agent": [
                    "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
                ],
                "Accept": [
                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
                ],
                "Accept-Language": [
                    "en-GB,en;q=0.5"
                ],
                "Accept-Encoding": [
                    "gzip, deflate, br"
                ],
                "Dnt": [
                    "1"
                ],
                "Upgrade-Insecure-Requests": [
                    "1"
                ],
                "Cache-Control": [
                    "max-age=0"
                ]
            },
            "tls": {
                "resumed": true,
                "version": 772,
                "ciphersuite": 4865,
                "proto": "h2",
                "proto_mutual": true,
                "server_name": "gateway.host"
            }
        },
        "common_log": "xxxxxxxxxxx - - [29/Mar/2020:17:07:49 +0000] \"GET / HTTP/2.0\" 200 0",
        "latency": 0.031736535,
        "size": 0,
        "status": 200,
        "resp_headers": {
            "Date": [
                "Sun, 29 Mar 2020 17:07:49 GMT"
            ],
            "Server": [
                "Caddy",
                "Caddy"
            ],
            "Content-Length": [
                "0"
            ]
        }
    }
    gateway-caddy | {
        "level": "info",
        "ts": 1585501669.1829925,
        "logger": "http.log.access.log1",
        "msg": "handled request",
        "request": {
            "method": "GET",
            "uri": "/",
            "proto": "HTTP/2.0",
            "remote_addr": "xxxxxxxx:xxxx",
            "host": "gateway.host",
            "headers": {
                "Accept-Encoding": [
                    "gzip, deflate, br"
                ],
                "Dnt": [
                    "1"
                ],
                "Upgrade-Insecure-Requests": [
                    "1"
                ],
                "Cache-Control": [
                    "max-age=0"
                ],
                "Te": [
                    "trailers"
                ],
                "User-Agent": [
                    "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:74.0) Gecko/20100101 Firefox/74.0"
                ],
                "Accept": [
                    "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8"
                ],
                "Accept-Language": [
                    "en-GB,en;q=0.5"
                ]
            },
            "tls": {
                "resumed": true,
                "version": 772,
                "ciphersuite": 4865,
                "proto": "h2",
                "proto_mutual": true,
                "server_name": "gateway.host"
            }
        },
        "common_log": "xxxxxxxxxxx - - [29/Mar/2020:17:07:49 +0000] \"GET / HTTP/2.0\" 200 0",
        "latency": 0.031736535,
        "size": 0,
        "status": 200,
        "resp_headers": {
            "Content-Length": [
                "0"
            ],
            "Date": [
                "Sun, 29 Mar 2020 17:07:49 GMT"
            ],
            "Server": [
                "Caddy",
                "Caddy"
            ]
        }
    }

5. What I already tried:

Various directives, header experimentation - although I’m not entirely sure what I’m doing here so it’s more trial and error.

Interestingly, curl to https://service.host:28989 from the gateway machine absolutely fine, it returns the expected page data.
This seems to suggest access is fine and the caddy on service.host is handling the requests as expected. The gateway.host also works when reverse proxying to another service on service.host which is not behind the service.host caddy.

The service is http not https but this seems ok through a standard curl from gateway.host.

6. Links to relevant resources:

2

This usually means that the origin of the response wrote an empty response. Check to make sure that your localhost:1234 is actually returning a non-empty response for that request.

Note that Caddy’s reverse proxy carries all (well, all non-hop-by-hop) headers through by default, including the Host header, and some backends are sensitive to the Host header. Some expect it to be the outside-facing hostname, while others expect it to be its internal hostname. If your backend requires a Host of localhost:1234, then make sure that is what Caddy is sending. Enable debug mode to see more about proxied requests in your log:

{
    debug
}
3

Speedy!

Yes, I checked the localhost service by running
curl https://service.host:28989 from the gateway.host machine. The data returns complete.

I’ll try enabling debug (looks like a global option from your reply).

I did wonder if it was something related to host headers but i’m getting out of my comfort zone at header detail level :smiley:
Could I somehow force the service.host caddy to use it’s hostname in the header? Does that even make sense? haha

4

I mean, run curl -v localhost:1234, then run curl -v -H "Host: service.host:1234" localhost:1234 then run curl -v -H "Host: gateway.host:1234" localhost:1234 from the backend’s machine, and see if there are any differences.

5

Service is happy with any of the hosts, they all return the same.

I did notice something with another test though.

If I curl service.host:28989 i.e., go through the service host caddy, I see two log messages, a debug one for the http.handlers.reverse_proxy followed by an info one for http.log.access.

Now, if I then curl gateway.host I see only one log message, an info for http.log.access.

Furthermore, If I keep trying these the service.host caddy eventually stop logging anything to the console. It does seem to be working though, in that I can curl the service.host:28989 successfully.

Strange…

6

Ah, that might be because service.host is configured as service.host, thus expects the Host header to be service.host, rather than gateway.host.

Read about how addresses work in the Caddyfile: Caddyfile Concepts — Caddy Documentation

You might need to set the Host header to the upstream hostname at each proxy. Use header_up: reverse_proxy (Caddyfile directive) — Caddy Documentation

7

You got it, fantastic.

I added header_up Host {http.reverse_proxy.upstream.hostport} to both reverse proxy directives on the gateway and the service and it works perfectly.

Thanks for the help, much appreciated.

P.S. This is a useful use-case when you are stuck behind a Dual-Stack Lite connection.

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.