Calling the admin api from within a route route causes a deadlock

YourTechBud · January 31, 2021, 7:02pm

I am writing a caddy app which configures other apps like http. My caddy app is triggered via a http.handler module.

What I’m saying is my http.handler calls the admin API as a part of its logic. This results in some sort of a deadlock and the admin API does not return indefinitely.

The problem gets resolved if I call the admin API asynchronously in a goroutine.

Is there a better way to do this? It’s important for me to make this call synchronous.

matt · January 31, 2021, 7:43pm

See this discussion:

github.com/caddyserver/caddy

Modify Caddy config using API called from proxied server

opened 12:02AM - 26 Sep 20 UTC

closed 05:39AM - 26 Sep 20 UTC

anderspitman

discussion

I'm working on a service that receives JSON requests and uses them to make chang…es to my Caddy config (for dynamically setting up proxies). So basically I'm publicly (with auth) exposing a wrapped subset of Caddy's own API. However, I'm running into issues and I think it has something to do with the fact that my service is itself proxied by the Caddy instance it is modifying. If I try to POST a new Caddy route from within my service, it hangs for about 5 seconds, then the Caddy API request finally returns, but my original JSON client receives an empty response from Caddy. I'm guessing Caddy is killing the original request after some timeout. Is there a proper way to accomplish what I'm trying to do?

What’s happening is the admin endpoint is trying to shut down the server that is proxying the request to the admin endpoint through it, which is waiting for the admin endpoint to return, but the admin endpoint is waiting for the proxy to finish responding, which won’t respond until the admin endpoint does, which won’t until… you get the idea.

It will eventually finish, but forcefully after some time, and you won’t get a valid HTTP response. So you won’t know if the config change succeeded or not.

While you may proxy the admin endpoint, it is equipped to be accessed directly. If you do use a proxy, it should be a proxy that does not depend on config from the admin endpoint you’re trying to proxy.

I recommend accessing the admin endpoint directly.

It’s impossible.

If you want a valid HTTP response, something has to give somewhere: you will have to introduce asynchronicity into the system.

This works, but we do not do this for you because it would require dropping some information on the floor: either we respond with 202 Accepted, meaning, “We’ll try to load this config. You’ll have to make another request later to find out if it worked.” The only way to get an error detail would be in the logs. Or, we can get that information but risk resource depletion. Alternatively still, we can drop the synchronization entirely and risk memory corruption (race condition).

Be aware that with a goroutine you still have to synchronize any return value (error output) if you want that (which you need, if you want to write a useful HTTP response other than 202), and you can also deplete resources if you create goroutines unbounded.

My highest recommendation is to not proxy the admin endpoint through the server that is controlled by the admin endpoint.

Otherwise, you’ll have to implement asynchronicity in the way that is best for your specific use case. There is no good “right” answer here, so we do not do it for you.

YourTechBud · February 1, 2021, 3:17am

I see. So are you suggesting I go for a Sidecar pattern here?

My http.Handler isn’t doing a transparent proxy here. It takes config in one format, validates it, then generates a caddy config format and applies it to caddy.

My downstream client isn’t caddy aware and it cannot be.

YourTechBud · February 1, 2021, 3:30am

After thinking about this for 5 more minutes, I’ve come up with a feature request.

Can we make the admin API pluggable as well? As in you let me inject handlers exposing custom endpoints on the admin server. It need not be super flexible like caddy. It could be as simple as /apply/HANDLER_NAME. Then in my handler, give me the methods i can call to set the config after doing my checks and transformations.

This way I can make my own variations of /load

matt · February 1, 2021, 5:03am

That’s basically what a config adapter does. Why not make your code into one of those? Then you can use the existing /load endpoint with your format’s Content-Type header.

It already is. The /load endpoint is a plugin:

github.com

caddyserver/caddy/blob/e2940c8c03818f1227aa2c103e05fb5664635e11/caddyconfig/load.go#L30-L62


      
          func init() {
          	caddy.RegisterModule(adminLoad{})
          }
          
          // adminLoad is a module that provides the /load endpoint
          // for the Caddy admin API. The only reason it's not baked
          // into the caddy package directly is because of the import
          // of the caddyconfig package for its GetAdapter function.
          // If the caddy package depends on the caddyconfig package,
          // then the caddyconfig package will not be able to import
          // the caddy package, and it can more easily cause backward
          // edges in the dependency tree (i.e. import cycle).
          // Fortunately, the admin API has first-class support for
          // adding endpoints from modules.
          type adminLoad struct{}
          
          // CaddyModule returns the Caddy module information.
          func (adminLoad) CaddyModule() caddy.ModuleInfo {
          	return caddy.ModuleInfo{
          		ID:  "admin.api.load",

This file has been truncated. show original

Does that help?

YourTechBud · February 1, 2021, 8:29am

This does sound like what I need. I’m still not sure on how and where can I register a custom adapter…

Edit:

I see there is a caddyconfig.RegisterAdapter i can use to add a adapter in my init() function. Then i can pass the content type header to the /load endpoint with my adapter name. My Adapt function will have to return the entire json payload caddy needs.

Am I understanding this right?

YourTechBud · February 1, 2021, 2:14pm

So this is what I ended up doing finally.

I made a module in the admin.api namespace. This lets me create custom endpoints. My endpoints then call methods on a singleton struct (since it is stateful) I’ve made on the global level which internally calls caddy.Load. This eliminates the need to right any asynchronous code since the request is being handled by the admin module.

I do have some complains though.

There is no caddy.ReadConfig method to read existing config. I see there is a method which seems to do the same but that isn’t exposed.
There is no way supply custom config to the admin module. Due to this, I have to rely on env variables to load an external config file. Also, the admin.api modules should have access to the AdminConfig.

Both these issues can we resolved easily in my opinion:

Simply expose caddy.readConfig to resolve the first issue.
admin.api can optionally implement a caddy.Provisioner interface from where they can get access to the AdminConfig.
Add a Extras map in the AdminConfig to accommodate extra config.

matt · February 1, 2021, 4:49pm

Yep! That’s it.

Ok so I read all that, but it still seems like way more work than is necessary. A config adapter is all you need to write that will transform any config format into Caddy JSON. You can use the existing /load endpoint for that. You don’t need to write your own admin endpoint handler. I want to clear this up before we continue because it seems like something you’re doing is making things harder than is necessary.

YourTechBud · February 1, 2021, 5:27pm

As per my requirements, I need to have a custom endpoint in the form of /v1/config/{type} to accept my configuration. Also, my downstream client cannot set a custom value in the Content-Type header.

Like caddy, my users can update specific parts of my configuration. That is why I need a stateful config adapter.

matt · February 1, 2021, 5:52pm

That’s unfortunate. It will make things a little tricker but not much. You should file a bug with your backend and see if they will fix that.

Just perform a GET /config/ request.

I am not sure what you mean by this?

Why is that needed/relevant?

YourTechBud · February 1, 2021, 6:01pm

That’s unfortunate. It will make things a little tricker but not much. You should file a bug with your backend and see if they will fix that.

I’m building a replacement for an existing tool so can’t do much here.

Just perform a GET /config/ request.

That’s what I’m doing rn. Seems like a round about way of doing it though. Would be great if I could access the config via a fn call.

I am not sure what you mean by this?

There are static parameters my config adapter needs. Things like configFormat (json | yaml). In order words, the behavior of the adapter needs to be configurable. There is no way to provide such information to my config adapter.

Just to be clear, I’m calling my admin.api module an adapter. But I guess the same thing holds true for caddyconfig.Adapter.

matt · February 1, 2021, 6:55pm

I can consider it, but in the meantime, your stateful config adapter could also keep it, right? It already knows all the JSON that’s going to Caddy, so…

Ah, gotcha. Yeah, well, I think that makes sense though: config adapters don’t have any JSON config exposed because their purpose is to produce the JSON config, not be produced by it. In other words, JSON is their output, not their input.

YourTechBud · February 2, 2021, 2:44am

I can consider it, but in the meantime, your stateful config adapter could also keep it, right? It already knows all the JSON that’s going to Caddy, so…

Yes. But things like the logging & admin config has to be the same. So i need to retrieve everything at least once since my module doesn’t really have that information on startup. And I can’t use your APIs to modify parts of the config since I’ll end up in a deadlock again.

Hitting the API to get the config works for now.

Ah, gotcha. Yeah, well, I think that makes sense though: config adapters don’t have any JSON config exposed because their purpose is to produce the JSON config, not be produced by it. In other words, JSON is their output, not their input.

I totally agree with you. But if caddy wants to be a mechanism to build apps in, it needs to make the admin api more configurable. I really like the admin.api approach since it lets me make my custom endpoints. All I need is something like the caddy.Provisoner support for admin.api from where I can access the admin config.

In a real world use case, people would like to plugin a store to distribute the config throughout a cluster. The config for the store and things like that should be, in my opinion, in the admin config. It cannot be an app since that limits what i can do due to the deadlock issue.

Also, i noticed, we setup the admin module before the logging Module. Not sure why that’s the case, but I can’t seem to get a handle of my custom loggers in my admin.api module. I’ll create another topic for this

francislavoie · February 2, 2021, 5:21am

I didn’t see Matt link this yet but I think it’s relevant, so I will; the 3rd feature from this recently merged PR seems relevant to you:

github.com/caddyserver/caddy

admin: Secure socket for remote management

caddyserver:master ← caddyserver:remote-admin

opened 10:32PM - 25 Jan 21 UTC

mholt

+862 -214

This PR adds 3 separate, but very related features: 1. Automated server ident…ity management 2. Remote administration over secure connection 3. Dyanmic config loading at startup ## 1. Automated server identity management How do you know you're connecting to the server you think you are? How do you know the server connecting to you is the server instance you think it is? Mutually-authenticated TLS (mTLS) answers both of these questions. Using TLS to authenticate requires a public/private key pair (and the peer must trust the certificate you present to it). Fortunately, Caddy is really good at managing certificates by now. We tap into that power to make it possible for Caddy to obtain and renew its own identity credentials, or in other words, a certificate that can be used for both server verification when clients connect to it, and client verification when it connects to other servers. Its associated private key is essentially its identity, and TLS takes care of possession proofs. This configuration is simply a list of identifiers and an optional list of custom certificate issuers. Identifiers are things like IP addresses or DNS names that can be used to access the Caddy instance. The default issuers are ZeroSSL and Let's Encrypt, but these are public CAs, so they won't issue certs for private identifiers. Caddy will simply manage credentials for these, which other parts of Caddy can use, for example: remote administration or dynamic config loading (described below). A bare-bones config might look like this: ```json { "admin": { "identity": { "identifiers": [ "123.123.123.123", "example.com", "127.0.0.1", "localhost" ], "issuers": [ { "module": "acme", "ca": "https://my-acme-server.example.com/", "trusted_roots_pem_files": ["my-acme-root.crt"] } ] } } } ``` Here, Caddy is told that its identities are those IP addresses and DNS names. It then will use your custom ACME server with a custom root certificate (to trust when connecting to it) to get certificates for those identifiers. Note that in this case, your CA would have to issue certs for localhost and 127.0.0.1, which most CAs don't do, since they can't be verified if they are remote. ## 2. Remote administration over secure connection This feature adds generic remote admin functionality that is safe to expose on a public interface. - The "remote" (or "secure") endpoint is optional. It does not affect the standard/local/plaintext endpoint. - It's the same as the [API endpoint on localhost:2019](https://caddyserver.com/docs/api), but over TLS. - TLS cannot be disabled on this endpoint. - TLS mutual auth is required, and cannot be disabled. - The server's certificate _must_ be obtained and renewed via automated means, such as ACME. It cannot be manually loaded. - The TLS server takes care of verifying the client. - The admin handler takes care of application-layer permissions (methods and paths that each client is allowed to use).\ - Sensible defaults are still WIP. - Config fields subject to change/renaming. Here's a basic example config, that I will explain: ```json { "admin": { "identity": { "identifiers": ["example.com"] }, "remote": { "access_control": [ { "public_keys": ["base64-encoded DER certificate"] } ] } } } ``` Explanation: - First we configure identity management. We tell Caddy that its identifier is `example.com`, so it will try to obtain and renew a certificate for that domain. By default, it will use publicly-trusted CAs. This is OK for DNS names that are properly configured. _Identity management is required when enabling remote administration, otherwise the server cannot present a TLS certificate to the client and secure the connection._ - We've enabled a secure admin endpoint at its default address **`:2021`** (you can customize it with `"listen": "..."` just like the regular admin endpoint) - note that the default address is not bound to a local interface, so it can be accessed remotely. - A single public key is then added to the ACL. Only the sole bearer of the associated private key is allowed unrestricted access of the API. We can also restrict different clients/users as for methods and paths they are allowed to access: ```json { "public_keys": ["base64-encoded DER certificate"], "permissions": [{ "paths": ["/id/foo/"], "methods": ["GET"] }] } ``` All the users specified in `public_keys` will be allowed to access all paths in the API starting with `/id/foo/` using only the `GET` method. As you can see, you can specify multiple paths and methods, and multiple groups of them, per group of public keys. Other advanced functionality is a bit limited because we cannot import any Caddy modules: they all import this package instead! So, we cannot import the `caddyhttp` or `caddytls` packages and take advantage of their advanced routing or security logic. The admin controls are relatively simple, but I imagine this should be more than enough...? Caddyfile config can probably be added later. ## 3. Dynamic config loading at startup Since this feature was planned in tandem with remote admin, and depends on its changes, I am combining them into one PR. Dynamic config loading is where you tell Caddy how to load its config, and then it loads and runs that. First, it will load the config you give it (and persist that so it can be optionally resumed later). Then, it will try pulling its _actual_ config using the module you've specified (dynamically loaded configs are _not_ persisted to storage, since resuming them doesn't make sense). This PR comes with a standard config loader module called `caddy.config_loaders.http`. Here's how it looks: ```json { "admin": { "config": { "load": { "module": "http", "url": "https://example.com/my_caddy_config.json" } } } } ``` Caddy will download the config at the given URL and run it. You can also configure authentication -- both client and server -- to ensure you get only trusted configs. If you add this to your config: ``` "tls": { "use_server_identity": true } ``` then Caddy will use the configured identity (explained above) as a client certificate to present to the server it is connecting to. In this case, identity management must also be configured.

matt · February 2, 2021, 6:17pm

I’m still not understanding… the admin and logging config are also part of the JSON config, meaning, your adapter will have access to that, too. Just feed all config through the adapter, right?

You can already implement Provisioner for your admin.api module, ah, but you mean that you want to be able to access the current Admin config from within the provisioning step? We already expose Storage, Logging, and Apps through the caddy.Context, so I guess we can expose the Admin config. Please file an issue so we can track that? Should be easy to do.

I recently revisited this, actually, in the PR that Francis just linked to above. I even switched it around momentarily while I was trying some things. I switched it back because it turns out I didn’t need to put logging first, but if you have a need for it, then I guess we should look at that too. Please open an issue and I’ll investigate further. Thanks!

To clarify, I’m open to making changes that export new functions, but I do that only carefully, after considering other possibilities, because I want to make sure they satisfy real use cases first.

YourTechBud · February 2, 2021, 10:34pm

I’m still not understanding… the admin and logging config are also part of the JSON config, meaning, your adapter will have access to that, too. Just feed all config through the adapter, right?

I was just explaining why i need to pull the json config again. It because my adapter only generates JSON config for the http app. The rest needs to be carry forwarded as is.

You can already implement Provisioner for your admin.api module, ah, but you mean that you want to be able to access the current Admin config from within the provisioning step? We already expose Storage, Logging, and Apps through the caddy.Context, so I guess we can expose the Admin config. Please file an issue so we can track that? Should be easy to do.

That would be super helpful. I’ll raise an issue right away. Also, can you make the admin config extendable as well while you at it?

I recently revisited this, actually, in the PR that Francis just linked to above. I even switched it around momentarily while I was trying some things. I switched it back because it turns out I didn’t need to put logging first, but if you have a need for it, then I guess we should look at that too. Please open an issue and I’ll investigate further. Thanks!

I wanted the logging level for my code to be debug while the rest of caddy could use info. So when I do logs in my admin.api, i couldn’t see my debug logs. I suspect its because logging wasn’t setup till then.

I also believe its a good practice to setup logging first unless there is some issue preventing it.

To clarify, I’m open to making changes that export new functions, but I do that only carefully, after considering other possibilities, because I want to make sure they satisfy real use cases first.

I totally understand this. I’m open to coming on a quick call to discuss my use case. You can then decide if you want that to be within the scope of Caddy.

matt · February 2, 2021, 10:53pm

There won’t be any other config though, unless it is put into the config by your adapter. Or if config is otherwise bypassing your adapter (but that’s in your control).

In what way? It already is, as you’ve discovered, you have a custom endpoint for it.

My priority was to set up the admin endpoint first, so that if anything goes wrong, the admin endpoint can be used to change config and recover from problems. If you’ve got a caddy instance running with no admin endpoint, it’s just dead in the water. I’ll have to think about if logging is worth risking that.

Just type it out here when you have a chance – that’ll be easiest!

YourTechBud · February 2, 2021, 11:28pm

There won’t be any other config though, unless it is put into the config by your adapter. Or if config is otherwise bypassing your adapter (but that’s in your control).

But, I don’t know what it is unless I read it.

Wait. I think i know what the mismatch is about. I’m starting the server with a default config caddy.json file. So my adapter doesn’t need know what the logging config and stuff like that is. So the op guy can start Caddy with any default config (to configure logging and storage), my adapter takes ownership of just the http app and nothing else.

In what way? It already is, as you’ve discovered, you have a custom endpoint for it.

My application needs setup parameters which I cannot take over the API. So we have another config file where all of that goes. We have two config files. One is an initial caddy.json with the initial logging and storage config, the other is a myapp.yaml which has my setup parameters. Right now i have to start caddy with an env variable (with the location of my setup file) which my adapter reads to load myapp.yaml cause I could not find any way to provide such information to my adapter.

Maintaining two files is painful. It would be great if I could embed my config in the caddy.json we have. I can’t put all of that config in a custom app since that will cause the admin module to depend on an app which sounds wrong. The only logical place to embed this config would be in the admin config of caddy.json.

My priority was to set up the admin endpoint first, so that if anything goes wrong, the admin endpoint can be used to change config and recover from problems. If you’ve got a caddy instance running with no admin endpoint, it’s just dead in the water. I’ll have to think about if logging is worth risking that.

That makes sense since logging itself can be configured via the API. I can do without configurable logging in my admin module.

But I don’t see myself changing logging config at runtime.

Well, I’ll leave that upto you.

matt · February 3, 2021, 3:29pm

Ahhh okay, so some config is bypassing your adapter. Thanks for clearing that up.

Would it be easier if Caddy could pull its config at startup? The latest commits on master make this possible (as Francis linked to earlier, I just want to be sure you saw it):

github.com/caddyserver/caddy

admin: Secure socket for remote management

caddyserver:master ← caddyserver:remote-admin

opened 10:32PM - 25 Jan 21 UTC

mholt

+862 -214

This PR adds 3 separate, but very related features: 1. Automated server ident…ity management 2. Remote administration over secure connection 3. Dyanmic config loading at startup ## 1. Automated server identity management How do you know you're connecting to the server you think you are? How do you know the server connecting to you is the server instance you think it is? Mutually-authenticated TLS (mTLS) answers both of these questions. Using TLS to authenticate requires a public/private key pair (and the peer must trust the certificate you present to it). Fortunately, Caddy is really good at managing certificates by now. We tap into that power to make it possible for Caddy to obtain and renew its own identity credentials, or in other words, a certificate that can be used for both server verification when clients connect to it, and client verification when it connects to other servers. Its associated private key is essentially its identity, and TLS takes care of possession proofs. This configuration is simply a list of identifiers and an optional list of custom certificate issuers. Identifiers are things like IP addresses or DNS names that can be used to access the Caddy instance. The default issuers are ZeroSSL and Let's Encrypt, but these are public CAs, so they won't issue certs for private identifiers. Caddy will simply manage credentials for these, which other parts of Caddy can use, for example: remote administration or dynamic config loading (described below). A bare-bones config might look like this: ```json { "admin": { "identity": { "identifiers": [ "123.123.123.123", "example.com", "127.0.0.1", "localhost" ], "issuers": [ { "module": "acme", "ca": "https://my-acme-server.example.com/", "trusted_roots_pem_files": ["my-acme-root.crt"] } ] } } } ``` Here, Caddy is told that its identities are those IP addresses and DNS names. It then will use your custom ACME server with a custom root certificate (to trust when connecting to it) to get certificates for those identifiers. Note that in this case, your CA would have to issue certs for localhost and 127.0.0.1, which most CAs don't do, since they can't be verified if they are remote. ## 2. Remote administration over secure connection This feature adds generic remote admin functionality that is safe to expose on a public interface. - The "remote" (or "secure") endpoint is optional. It does not affect the standard/local/plaintext endpoint. - It's the same as the [API endpoint on localhost:2019](https://caddyserver.com/docs/api), but over TLS. - TLS cannot be disabled on this endpoint. - TLS mutual auth is required, and cannot be disabled. - The server's certificate _must_ be obtained and renewed via automated means, such as ACME. It cannot be manually loaded. - The TLS server takes care of verifying the client. - The admin handler takes care of application-layer permissions (methods and paths that each client is allowed to use).\ - Sensible defaults are still WIP. - Config fields subject to change/renaming. Here's a basic example config, that I will explain: ```json { "admin": { "identity": { "identifiers": ["example.com"] }, "remote": { "access_control": [ { "public_keys": ["base64-encoded DER certificate"] } ] } } } ``` Explanation: - First we configure identity management. We tell Caddy that its identifier is `example.com`, so it will try to obtain and renew a certificate for that domain. By default, it will use publicly-trusted CAs. This is OK for DNS names that are properly configured. _Identity management is required when enabling remote administration, otherwise the server cannot present a TLS certificate to the client and secure the connection._ - We've enabled a secure admin endpoint at its default address **`:2021`** (you can customize it with `"listen": "..."` just like the regular admin endpoint) - note that the default address is not bound to a local interface, so it can be accessed remotely. - A single public key is then added to the ACL. Only the sole bearer of the associated private key is allowed unrestricted access of the API. We can also restrict different clients/users as for methods and paths they are allowed to access: ```json { "public_keys": ["base64-encoded DER certificate"], "permissions": [{ "paths": ["/id/foo/"], "methods": ["GET"] }] } ``` All the users specified in `public_keys` will be allowed to access all paths in the API starting with `/id/foo/` using only the `GET` method. As you can see, you can specify multiple paths and methods, and multiple groups of them, per group of public keys. Other advanced functionality is a bit limited because we cannot import any Caddy modules: they all import this package instead! So, we cannot import the `caddyhttp` or `caddytls` packages and take advantage of their advanced routing or security logic. The admin controls are relatively simple, but I imagine this should be more than enough...? Caddyfile config can probably be added later. ## 3. Dynamic config loading at startup Since this feature was planned in tandem with remote admin, and depends on its changes, I am combining them into one PR. Dynamic config loading is where you tell Caddy how to load its config, and then it loads and runs that. First, it will load the config you give it (and persist that so it can be optionally resumed later). Then, it will try pulling its _actual_ config using the module you've specified (dynamically loaded configs are _not_ persisted to storage, since resuming them doesn't make sense). This PR comes with a standard config loader module called `caddy.config_loaders.http`. Here's how it looks: ```json { "admin": { "config": { "load": { "module": "http", "url": "https://example.com/my_caddy_config.json" } } } } ``` Caddy will download the config at the given URL and run it. You can also configure authentication -- both client and server -- to ensure you get only trusted configs. If you add this to your config: ``` "tls": { "use_server_identity": true } ``` then Caddy will use the configured identity (explained above) as a client certificate to present to the server it is connecting to. In this case, identity management must also be configured.

Anyway, your use case is definitely a unique one! I’ll try to accommodate best we can. Please feel free to open issues for any remaining concerns and that way we won’t forget to look at them.

YourTechBud · February 3, 2021, 4:28pm

I saw that PR. But i don’t have a external server the config can be pulled from. All i need is ability to add custom parameters in the admin config and make it accessible in modules through caddy.Context. Both seem to be small changes in my opinion.

Will raise an issue for it.