Best way to add CDN for reverse proxy with custom domains?

1. Caddy version (caddy version):

Caddy 2.0.0

2. How I run Caddy:

Pre-built digital ocean droplet

a. System environment:

Ubuntu 18.04

b. Command:

systemctl start caddy

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

Pretty straightforward reverse proxy for multiple custom domains:

{
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [":443"],
          "routes": [
            {
              "handle": [
                {
                  "handler": "subroute",
                  "routes": [
                    {
                      "handle": [
                        {
                          "handler": "reverse_proxy",
                          "headers": {
                            "request": {
                              "set": {
                                "Host": ["{http.reverse_proxy.upstream.host}"],
                                "X-Forwarded-Host": ["{http.request.host}"]
                              }
                            }
                          },
                          "transport": {
                            "protocol": "http",
                            "tls": {}
                          },
                          "upstreams": [
                            {
                              "dial": "notion2site-renderer.onrender.com:443"
                            }
                          ]
                        }
                      ]
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "masterbranch.dev",
                    "lownocode.dev",
                    "demo.notion2site.com",
                    "saas-journey.com",
                    "test.saas-journey.com",
                    "test2.saas-journey.com"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    }
  }
}

3. The problem I’m having:

I have a few general questions around production concerns for this type of reverse proxy use case.

My customers point their custom domains to my caddy server’s static IP address on Digital Ocean, and caddy reverse proxies this traffic to an upstream Next.js webserver (currently hosted on render.com) to render their website’s content dependent on the original origin’s Host header.

Everything is working great, and I have to say that Caddy is amazing and I’ll definitely be using it for multiple projects in the future. :100:

A few questions I have:

  1. Is there a simple way to place a CDN in front of this caddy server? It would have to have a static IP address so my customers can use DNS A records to set up their domain’s config.

  2. If the CDN in front of caddy isn’t easily doable, I could use a CDN in-between Caddy and my upstream Next.js server. My main goal is to enable aggressive caching of the reverse proxied responses. Any advice on approaches to this with caddy?

Ideally the solution would use an edge CDN for caching but if all requests have to go through the Caddy server, then this negates the benefits of an edge CDN like the ones provided by Cloudflare or Vercel…

  1. Are there any resources that could give me an idea of how much traffic this single Caddy server can handle by just scaling its resources vertically? I’m not sure how comfortable I am delegating this type of platform’s core architecture to a single caddy server, but I remember reading somewhere on these forums that it scales surprisingly well – I’d just love some data or live production examples to back that up.

  2. With some trial and error, I was able to get this use case working pretty easily – kudos on the docs :muscle: The one area that I couldn’t get working easily was logging which is especially useful during debugging. Having some examples of how to configure the logging in Caddy v2 for the simple 95% use case of logging during debugging without all the bells & whistles would be super helpful.

Thank you for your time. :smile:

1 Like

Welcome Travis! Thanks for sharing your use case. Always helpful to know that.

The same concerns as with any other web server apply with the addition of the ACME challenges: if Caddy is to solve them, it needs the external ports 80 and 443 forwarded to it. So if Caddy is behind a CDN or any kind of proxy or connection terminator, you will need to make sure that ACME challenge HTTP requests and ACME challenge TLS connections go directly to Caddy; do not pass go, do not terminate TLS. Most CDNs should have a way to enable that.

There’s a WIP cache plugin: GitHub - caddyserver/cache-handler: Distributed HTTP caching module for Caddy - however, unless the community comes together to finish building it (I can supervise), best bet is for a company to hire Ardan Labs to develop it: https://caddyserver.com/business (they work in close contact with me for Caddy dev).

Well, Google, Netflix, and Cloudflare all use Go in production. I don’t have any hard numbers for you though. You’ll have to experiment and let us know. If you encounter any issues, do a profile and let’s optimize it. Unfortunately most people are too anxious to run useless benchmarks that test too many stacks and we can’t improve anything from that, since they don’t reveal any useful information. We need actual memory and CPU profiles for very specific loads.

Thanks, we don’t hear a lot of good things about our docs, but we work hard on them.

Just set the default log’s level to DEBUG. There’s not a lot of debug logs yet but we do emit URI rewrites and reverse proxy requests in the debug logs, which helps a majority of cases. We can add more if we find places where it would be useful. My recommendation with logging is to just let Caddy log anything and everything out to a single place and then you can aggregate and process them with proper tooling.

3 Likes

Thanks @matt for the awesome answers – it’s really appreciated when one of the project leads takes the time to answer community questions.

I ended up placing my Vercel Next.js server upstream from Caddy which is working really well so I didn’t need to worry about putting another CDN in front of Caddy.

You may find these lighthouse performance scores and my use case in general pretty interesting. This is all for notion2site which isn’t 100% ready to launch yet but it’s getting close. This SaaS product is powered by my startup, Saasify, and we’re definitely intending to use Caddy for a slew of similar SaaS products in the future.

Well, Google, Netflix, and Cloudflare all use Go in production.

Yeah, I’m less concerned about Go as that’s a known entity and rather just how caddy holds up on a single machine versus something like load-balancing a set of nginx proxies.

Thanks, we don’t hear a lot of good things about our docs, but we work hard on them.

Yeah; I think the biggest thing missing for me is concrete examples. The community forum helped a lot in this regard as I was able to pull from people’s previous trial & error to get most of a config together. One suggestion would be to have a set of templates for different standard use cases that are linked to from your various docs about the individual properties. This is something that Cloudflare Workers has done a great job of with their template gallery.

Just set the default log’s level to DEBUG . There’s not a lot of debug logs yet but we do emit URI rewrites and reverse proxy requests in the debug logs, which helps a majority of cases. We can add more if we find places where it would be useful. My recommendation with logging is to just let Caddy log anything and everything out to a single place and then you can aggregate and process them with proper tooling.

Honestly, because I was running a Caddy droplet on digital ocean I had no idea how to even view the logs since I wasn’t running Caddy directly. It was a total black box for the first few days experimenting with things and systemctl didn’t provide any default logs or output. It took me a long time to realize I could stop the caddy daemon and run it directly to view stdout at which point several problems were immediately obvious.

The one other piece of feedback I have is that the whole Caddyfile vs JSON formatting is really, really frustrating especially the fact that there isn’t a 1:1 mapping between them. I know I can start from a Caddyfile and use caddy adapt to get a JSON representation but there’s no way to go back which is a PITA for debugging.

I know this is a stretch, but honestly, this is the single most important thing that Cloudflare workers got right – the fact that it’s constrained but programmable means that I don’t have to understand the various intricacies of your config formats as long as I understand the primitives. I’d pay serious $$ for something programmable like CF workers via WASM / V8 that had the functionality that Caddy provides.

I view this as very analogous to how the GPU pipeline world changed dramatically once programmable shaders were introduced. Previously, you could just set various config settings and hope for the best which is pretty similar to how Nginx, Caddy, and most CDNs operate. The truly innovative thing that programmable shaders did for GPU programming is the same thing that CF workers have done for cloud networking – making them more much more accessible and extensible in a way that just isn’t possible with static configs.

Because ultimately the most difficult part imho of working with Caddy or Nginx is understanding the configs and trial & error with these esoteric DSLs. Even though I think you’ve done a great job with Caddy’s config DSL, it’s still will never be as easy as using an existing language.

Anyhow, I hope this feedback is helpful. Thanks again for your thorough answers and for this great product :100:

1 Like

Thanks for the feedback.

This is exactly what our community wiki is for. We even link to it from the top of the docs nav.

The welcome page tells you how to view the logs. (It’s the same way you view logs for any other Linux service. Not really a Caddy-specific thing. It’s more of a “how you use Linux computers” kind of thing.)

I mean, you or anyone is welcome to write a reverser. If it’s that big of a pain, then just write one yourself and share it with us!

Frankly I don’t recommend using the Caddyfile at all when you’re programming your web serve. Just use the JSON, then you don’t have to worry about the Caddyfile or its mapping at all.

The JSON is the absolute truth. Debug your config in the JSON format.

1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.