Programmatically update config from a Go program

1. Caddy version:

2.6.2

2. How I installed, and run Caddy:

a. System environment:

pacman -Suy caddy
Manjaro Linux 22.0.4 x86

b. Command:

caddy run

c. Service/unit/compose file:

n/a

d. My complete Caddy config:

{
	auto_https off
}

http://localhost:2000 {
	reverse_proxy localhost:8080
}

http://customer1.localhost:2000 {
	forward_auth localhost:8080 {
		uri /init
	}

	reverse_proxy localhost:40943
}

(note that all the localhost stuff and disabling of https is because I’m developing still)

3. The problem I’m having:

Each customer of ours has their own subdomain. Customers can onboard themselves. The service running at port 8080 handles that. That same service also handle auth (via a HTTP header, irrelevant now).

What I want to do is that the service running at 8080 adds new site blocks. This is actually pretty straightforward. I just make a HTTP POST request to the relevant config endpoint. Something like this:

curl -H 'Content-Type: application/json' -v -d @new_domain.json localhost:2019/config/apps/http/servers/srv0/routes

The question is about the JSON document needing to be POST’ed. A site block with merely one reverse_proxy in it translates to a relatively simple JSON document. (I ‘converted’ the initial Caddyfile into JSON by starting Caddy with that file and making a GET request to the config endpoint of Caddy.)

But as soon as the forward_auth directive comes into play the JSON document balloons:

$ curl -v localhost:2019/config/apps/http/servers/srv0|jq
{
  "automatic_https": {
    "disable": true,
    "skip": [
      "subdomain.localhost",
      "localhost"
    ]
  },
  "listen": [
    ":2000"
  ],
  "routes": [
    {
      "handle": [
        {
          "handler": "subroute",
          "routes": [
            {
              "handle": [
                {
                  "handle_response": [
                    {
                      "match": {
                        "status_code": [
                          2
                        ]
                      },
                      "routes": [
                        {
                          "handle": [
                            {
                              "handler": "headers",
                              "request": {}
                            }
                          ]
                        }
                      ]
                    }
                  ],
                  "handler": "reverse_proxy",
                  "headers": {
                    "request": {
                      "set": {
                        "X-Forwarded-Method": [
                          "{http.request.method}"
                        ],
                        "X-Forwarded-Uri": [
                          "{http.request.uri}"
                        ]
                      }
                    }
                  },
                  "rewrite": {
                    "method": "GET",
                    "uri": "/init"
                  },
                  "upstreams": [
                    {
                      "dial": "localhost:8080"
                    }
                  ]
                },
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "localhost:40943"
                    }
                  ]
                }
              ]
            }
          ]
        }
      ],
      "match": [
        {
          "host": [
            "subdomain.localhost"
          ]
        }
      ],
      "terminal": true
    },
    {
      "handle": [
        {
          "handler": "subroute",
          "routes": [
            {
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "localhost:8080"
                    }
                  ]
                }
              ]
            }
          ]
        }
      ],
      "match": [
        {
          "host": [
            "localhost"
          ]
        }
      ],
      "terminal": true
    }
  ]
}

4. Error messages and/or full log output:

n/a

5. What I already tried:

My issue with this is that my service is written in Go. I already had a ton of structs and now it becomes unwieldy to say the least (for example, the Handler sometimes has this and that field, other times not).

Before I bite the bullet, I wondered if there is a Better Way.
For example: since my service is written in Go and Caddy is too I figured I could reuse the parsing logic and structs from Caddy! But that doesn’t seem to be exposed. Or perhaps I could just POST Caddyfile snippets, but only the /load endpoint is able to consume Caddyfiles.

6. Links to relevant resources:

See text

You can generate caddyfile and then post to /adapt endpoint to get a json result for your new config, and using jd to generate a diff with the current config, then apply the diff.

Currently I think just post to /reload is more simple because caddy doesn’t handle partial config update that well (to be improved later), having to stop running config then restart.

If your main sites and all customers’ sites follow the same forward_auth pattern, then you can see they all share the same subroute structure, you can post the same structure into routes array.

Exactly this. Any change to the config is a “full reload” (which is graceful) so there’s no practical difference between sending the entire config to Caddy vs pushing a change via /config. You can just use /load to push the Caddyfile.

But I’d suggest instead thinking about making your config not require any reloads at all. Why exactly do you need to reload? Why can’t you use On-Demand TLS to issue certs for the domains you need? Use dynamic upstreams module to decide where to proxy to (based on the domain or w/e). Write a custom module to do any other decision making if you need to.

1 Like

I’m not worried about the per domain certs. In fact when I get a wildcard cert for the TLD I’m good.

I think I need a reload because of two reasons:

  1. the directive reverse_proxy has a different argument for each customer, each upstream handles one, specific customer. Eg: reverse_proxy localhost:<THIS_IS_UNIQUE_PER_CUSTOMER>
  2. the site address in a site block is unique per customer.

But mentioning dynamic upstream is exactly why I posted my original question. Perhaps there’s something better. Wrt to the dynamic upstream; all upstreams in this case are the same right? Therefore unsuited in this case.

A custom module seems heavy handed?

That’s what a dynamic upstreams module would let you do. You’re using static upstreams now, but you could write a module that decides which upstream to use based on the hostname of the incoming request. See reverse_proxy (Caddyfile directive) — Caddy Documentation

You can have a single site block for all of them, if you’re using a wildcard.

No, dynamic upstreams modules can decide on the upstream based on the incoming request.

Not at all. It’s very simple to write plugins for Caddy. Extending Caddy — Caddy Documentation

In your case, for a dynamic upstreams module, write a module that implements UpstreamSource and is in the namespace http.reverse_proxy.upstreams.* as per Module Namespaces — Caddy Documentation

1 Like

I think I’m getting the hang of that custom module stuff. But I’m not clear on how to use it.
What I have so far is:

package single

import (
	"net/http"

	"github.com/caddyserver/caddy/v2"
	"github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy"
)

func init() {
	caddy.RegisterModule(Gizmo{})
}

type Gizmo struct {
	Pairs []Pair `json:"pairs,omitempty"`
}

type Pair struct {
	From string `json:"from,omitempty"`
	To   string `json:"to,omitempty"`
}

func (Gizmo) CaddyModule() caddy.ModuleInfo {
	return caddy.ModuleInfo{
		ID:  "http.reverse_proxy.upstreams.single",
		New: func() caddy.Module { return new(Gizmo) },
	}
}

func (Gizmo) GetUpstreams(r *http.Request) ([]*reverseproxy.Upstream, error) {
    // Some logic will come here: reading the domain from `r` looking that up in `Pairs` etc. Easy.
	return []*reverseproxy.Upstream{}, nil
}

I can’t use this in a Caddyfile. For that I need to register directives and whatnot. That’s fine. I want to use it with some HTTP POST/PUT/PATCH requests anyway.

I feel a bit of a dumbass asking this; but what now? I can’t figure out the format of the HTTP request to update the config.

In a semi-unrelated question: What on earth is going on here?

Yeah, you just need to call RegisterHandler to start. If it has no config, there’s nothing else to do, really. See Caddyfile Support — Caddy Documentation of course.

No, the point is that you won’t need to. You’ll only have one site block with this one reverse proxy.

*.example.com {
	reverse_proxy {
		dynamic your_module <config...>
	}
}

And your module’s job would just be to look at the r.Host and decide which upstream to send the request to. Query the mapping from your database or whatever.

The replacer is the thing that transforms placeholders in config to their final value. See Caddyfile Concepts — Caddy Documentation and Conventions — Caddy Documentation

Francis is right; to elaborate, it retrieves the replacer from the request context, which is a value that follows the request around the various parts of the code during the HTTP handling. We retrieve the replacer and then since Go is strongly-typed it comes back as the any type (interface{}), so we have to type-assert it to the specific type we know it to be in order to use it.

1 Like

You guys continue to be excellent!

I am hoping I can avoid doing external calls from this module. My datastore would be the Pairs field on the Gizmo struct. The other software I’m writing would be resilient to losing the in-memory data store.

I was toying around with the the upstreams and was able to do calls like these:
$ curl -H 'Content-Type: application/json' -v -d '[{"dial": "localhost:8082"}]' -X PATCH 'localhost:2019/config/apps/http/servers/srv0/routes/1/handle/0/routes/0/handle/0/upstreams'

And POST requests that appended to the list (as described in the API docs).

Isn’t that pattern workable? Or is it considered an anti pattern?

Thanks for clarifying that Replacer line!

No, you can totally do that. It just invokes a config reload, so be aware of that. (It’s graceful. But it can still interrupt really long-lived connections like websockets. For now.)

If you want to avoid a config reload, you can implement a dynamic upstreams module like Francis mentioned.

@haarts Just curious, what is your company/product? Would love to know where Caddy is being used.

The problem using a static list of upstreams is that then you have less control over which upstream is chosen for a given request, because it’s static. It doesn’t change based on the incoming request, it’s just always the same.

With dynamic upstreams you can look at a request, see the hostname is customer1.example.com and route it to foo:10001 and customer3.example.com to foo:10003 and so on. Which is what it sounds like you want in your case, since you have different upstreams per customer.

1 Like

Lucky for me I don’t like websockets at all and am not planning to use them. :slight_smile:

From the docs I understood that a dynamic reload entails. If this project is wildly successful, I expect a couple of thousand reloads comfortably spread over time. I’m not worried.

The product I’m going to use this for is https://disknotifier.com. I’m going to replace the current Rails backend with something else. Caddy is part of that ‘something else’.

The idea is as follows: every customer gets his own SQLite database and his own application binary (written in Go). The flow would be something like this:

  1. A HTTP request is received by Caddy.
  2. Caddy uses forward_auth to authenticate the request.
  3. The service pointed to by the forward_auth directive creates a new database + application binary (if necessary) and spins it up. It also updates the Caddy config to associate a domain with the just spun-up upstream.
  4. Subsequent requests to Caddy are forwarded to the now configured upstream.

This is completely counter to what is usually being done! I’m well aware. But this endeavor is a study into what is possible. In my day job, our company deals with incredibly sensitive data. We license a piece of software that is hosted on-premise with our clients. Having one client’s data in the same database as another is a no-go (regulation plays a big role). But on-premises software is understandably hard to sell. So we are looking at a hybrid form. The setup I’m experimenting with gives us the best of both worlds (I hope). We can even go as far as provisioning bare metal machines for each client!

The code required to do all this must be kept as small as possible. Less maintenance, less code to audit.
So I end up with three pieces of software:

  1. The customer application (we need that anyway)
  2. The service manager (the one behind forward_auth, currently 200 LOC)
  3. A custom Caddy module (I estimate 200 LOC as well)

I’m not thrilled about the Caddy module, to be honest, since this is such a specialized piece of software. However, the far above-average quality Caddy docs and the great support on this forum are enough to be OK with it. The experience of writing a Caddy module so far was great as well.

1 Like

I fully agree. Static upstreams are not usable in my scenario.

I got my little module to work (in principle)! I can now configure Caddy to add the single (terrible name, I know) reverse proxy like so:

curl -v -H 'Content-type: application/json' -d '{"dynamic_upstreams": {"pairs": [{"to": "foo", "from": "bar"}], "source": "single"}, "handler": "reverse_proxy"}'  localhost:2019/config/apps/http/servers/srv0/routes/1/handle/0/routes/0/handle/

I figured out the correct JSON format by adding a A dynamic reverse proxy and querying the config.

Adding subsequent pairs like so:

curl -v -H 'Content-type: application/json' -d '{"to": "foobar", "from": "baz"}'  localhost:2019/config/apps/http/servers/srv0/routes/1/handle/0/routes/0/handle/3/dynamic_upstreams/pairs/

Now I ‘just’ need to fill in the blanks, like getting the host of the request and do an efficient lookup in the Pairs slice.

1 Like

Once again, I really don’t think you need to push a config update to Caddy to make this work. Since your auth endpoint is doing the work or determining which customer it is already, it could just respond with an HTTP header which has the upstream address. So it could be as simple as this (with no custom module for Caddy in this case):

*.example.com {
	forward_auth localhost:8080 {
		uri /init
		copy_headers X-Upstream-Address
	}

	reverse_proxy {header.X-Upstream-Address}
}
1 Like

Only now I understand!!

That is brilliant. Thank you for taking the time to take it slow with me. Yes yes, I think you are absolutely right. Such elegance. I will try this at the first available opportunity.

2 Likes

That Just Works. Thank you!

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.