Load balancing Caddy -- am I doing it right?

Help
1

Dear community,

this is my first post, and I’d like to express my thank for caddy, a great software that has made my life much easier.

Note: it seems I cannot post some words so I had to keep changing the domain-related words to site1-domain-abc and the likes till it’s accepted. My apology for this.

Update: I updated the diagram to include the caddy config

1. The problem I’m having:

I have 1 server and 2 ISP lines at home. I would like to make the server available over those 2 ISP lines to increase availability. The setup looks as follows:

image
image1762×1972 228 KB

3. Caddy version:

v2.7.5

4. How I installed and ran Caddy:

I installed caddy from official Caddy apt repo

a. System environment:

Ubuntu 22.04, without docker

d. My complete Caddy config:

# on site1
site1 .domain-abc {
  reverse_proxy local-server
}

# on site2
site2 .domain-abc {
  reverse_proxy local-server
}

# on site
site.domain-abc {
  reverse_proxy {
    to site1-domain-abc
    to site2-domain-abc
    header_up Host {upstream_hostport}
    lb_policy round_robin
    fail_duration 3s
  }
}

This setup seems to work, however from the posts I found in the forum, the best practice seems to forward port 80 and 443 on the load balancer, and do TLS termination only at site1 and site2.

I have been using caddy for sometime for simple scenarios, but this is the first time I attempt to setup load balancing with https upstreams.

Ideally I would like to do TSL termination for site.domain-abc only, and get rid of site{1,2}.domain-abc completely. But I don’t know how to do that. Any hint to get started would be much appreciated.

5. Links to relevant resources:

1 Like
2

Caddy’s standard distribution doesn’t ship with a TCP-layer proxy, it only has an HTTP server & proxy. So if you want to avoid terminating TLS at your VPS, you’d need to use something like GitHub - mholt/caddy-l4: Layer 4 (TCP/UDP) app for Caddy to do so.

Your current setup seems fine though. I’m not sure you’ll notice any practical improvement from doing TCP proxying instead of HTTP. You can try both and compare though.

Keep in mind that if you proxy TCP then both your upstream servers will see site.domain-abc as the domain, and not 1/2. You’d probably need to ensure your Caddy instances share storage so that they can coordinate ACME issuance, otherwise you can run into problems. Or you could just use one Caddy instance at home, reachable by both lines, if you can connect your server to both lines?

1 Like
3

thank you for your reply.

Note: in the below I had to change site to website and my domain to example.com for the post to be accepted. Sorry again.

I use GoAccess as recommended in this forum to view the log.
On website{1,2}.example.com and local-server I see the domain of the requests as website{1,2}.example.com and not website.example.com as I would like to have.

I tried the following:

  • comment out the line header_up Host {upstream_hostport} on website.example.com (so that the domain stays website.example.com at upstream servers)
  • attempt to TLS terminate domain website.example.com on website1.example.com:
website1.example.com,
website.example.com {
  reverse_proxy local-server
}
  • similar for website2

Which looks quite questionable, since website1.example.com does not have the IP of website.example.com. And indeed it didn’t work. However, when I took away one upstream from website.example.com (so that it has exactly 1 upstream), it seems to work which I didn’t expect at all.

Edit: I updated the diagram to get rid of the forbidden words:

image
image1804×1952 244 KB

4

The original Host header gets passed down using the X-Forwarded-Host header. Things that need to know the original host should read that header.

Since you’re using multiple layers of proxies, you’ll need to configure trusted_proxies on your home servers, to trust the IP address of your VPS server, otherwise the X-Forwarded-* headers from the VPS will be ignored for security reasons. See Global options (Caddyfile) — Caddy Documentation

1 Like
5

I updated my setup as in this diagram:

image
image1944×2098 269 KB

and it seems to work well. It seems somewhat magical to me that Caddyfile on website1 & website2 have the address website.example.com; I thought it’s only valid if website.example.com resolves to the IP addrs of those servers, but it seems not the case.

I wonder what are the benefits of the way described in the wiki compared to this setup. For me it means:

  • setup shared storage for caddy data on website1 & website2; I can use NFS for it
  • create A records of website.example.com to point to IP addrs of website1 & website2
  • setup caddy-l4 on load balancer

which is somewhat challenging, but doable.

6

The site address is a “host matcher” (i.e. matches requests with the Host header having that value) plus enables that domain to have its certs managed if Automatic HTTPS is enabled.

What I’m confused about is it seems that you’re proxying from your VPS over HTTP, because you don’t have https:// in front of your upstream addresses (unless your screenshot is a lie). If that’s the case, then your home servers should be serving an HTTP->HTTPS redirect response for every request because they’re both configured to serve HTTPS (which is the default, unless http:// is prefixed on the site address).

What’s in the logs of your home servers? I can’t imagine Caddy successfully obtained a cert for your website domain, that would be weird.

In your image, you don’t have this anymore. Did you remove it?

What this does is set the Host header when proxying to the hostname of the upstream address. By default, the Host header is preserved, so website would be sent to both upstreams. With that line, website{1,2} would be sent as the hostname respectively, which would match the site address on the home servers.

7

my sincere apology for the mistake I made in the diagram. The upstream addresses have https:// in Caddyfile.

This time I copy&paste the content of Caddyfile from the servers and changed only the domain to example.com and IP of VPS to 1.2.3.4:

Caddyfile on home servers:

{
  log default {
    format json
    output file /var/log/caddy/default.log
  }
  servers {
    trusted_proxies static 1.2.3.4
  }
}
abc.example.com,
abc1.example.com {
  log {
    output file /var/log/caddy/abc.log
  }
  reverse_proxy testbox-dmz
}

Caddyfile on VPS:

{
  log default {
    format json
    output file /var/log/caddy/default.log
  }
}
abc.example.com {
  log {
    output file /var/log/caddy/abc.log
  }
  reverse_proxy {
    to https://abc1.example.com
    to https://abc2.example.com
    lb_policy round_robin
    fail_duration 3s
  }
}

I did some inspection on the home servers, the cert for abc1.example.com has been created but abc.example.com has not.

find /var/lib/caddy -name '*abc*'
/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/abc1.example.com
/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/abc1.example.com/abc1.example.com.crt
/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/abc1.example.com/abc1.example.com.json
/var/lib/caddy/.local/share/caddy/certificates/acme-v02.api.letsencrypt.org-directory/abc1.example.com/abc1.example.com.key
/var/lib/caddy/.local/share/caddy/ocsp/abc1.example.com-6ae4e711
/var/lib/caddy/.local/share/caddy/locks/issue_cert_abc.example.com.lock

The local server testbox-dmz is itself a caddy server, which serves a html page which shows all variables (placeholders) from caddy.

then I test by

watch 'curl -v  https://abc.example.com  2>&1    | rg client_ip\|hostport'

and it seems to work, the client_ip value is local IPs of the home servers, and hostport is abc.example.com as I’d like.

I removed the line

header_up Host {upstream_hostport}

so that the Host is not changed. Well it seems to work, but I don’t know it should work that way.

from the log on home server it looks like this:

{
  "level": "error",
  "ts": 1699962307.3860083,
  "logger": "tls.obtain",
  "msg": "will retry",
  "error": "[abc.example.com] Obtain: [abc.example.com] solving challenge: abc.example.com: [abc.example.com] authorization failed: HTTP 0  -  (ca=https://acme.zerossl.com/v2/DV90)",
  "attempt": 7,
  "retrying_in": 1200,
  "elapsed": 2535.383040493,
  "max_duration": 2592000
}

so it’s as you said, Caddy cannot obtain a cert for abc.example.com on the home servers. But I got no complain from curl and browsers. :thinking:

1 Like
8

You shouldn’t keep it this way because you risk eventually hitting rate limits. It’s sloppy.

So I suggest switching to overriding Host on the VPS and only accepting the exact host on the home servers. If your actual app needs Host to be the original value, you can fix that on the home servers similarly.

9

so I added:

on VPS:

header_up Host {upstream_hostport}

on home server:

header_up Host abc.examle.com

(and removed the address abc.examle.com)

This setup seems to work, but when I access abc1.example.com directly, the host is changed to abc.examle.com which seem incorrect. Is it possible to avoid this change when the request doesn’t come through the VPS?

10

That’s probably your app doing the redirect. You might be able to disable that redirect in the app, I dunno. Not a Caddy problem though.

11

sorry that my description was not clear.

on the home server abc1 I have

header_up Host abc.example.com

which changes the Host in the requests passed to upstream testbox-dmz to abc.example.com. So when I access https://abc1.example.com, the server testbox-dmz sees the requests with Host = abc.example.com. I wonder if it is possible to have something like this pseudo-code:

if request.headers."X-Forwarded-Host" == "abc.example.com" then
  header_up Host abc.example.com
12

Yeah, you can do this:

@trusted vars trusted_proxy true
request_header @trusted Host abc.example.com

This will override the Host header when the request is from a trusted proxy, and do nothing otherwise. So as long as you set trusted_proxies to only be the IP of your proxy, it should work fine.

In that config above, trusted_proxy is a special variable that’s set at the start of the request when the HTTP server detects the request as being from a trusted proxy.

1 Like
13

Thank you very much for your help. I really appreciate it.

14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.