Remote config loading does not return loaded config when making API requests

matthewpi · June 18, 2021, 7:12pm

1. Caddy version (`caddy version`):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

a. System environment:

RHEL 8.4

❯ uname -msr
Linux 4.18.0-240.15.1.el8_3.x86_64 x86_64

b. Command:

Caddy is ran as an unprivileged user using an executable in the same directory as the config and client certificates.

sudo setcap CAP_NET_BIND_SERVICE+ep <PATH_TO_CADDY>
./caddy run -config=caddy.json

c. Service/unit/compose file:

N/A

d. My complete Caddyfile or JSON config:

Local Config:

{
	"admin": {
		"listen": "127.0.0.1:2020",

		"config": {
			"load": {
				"module": "http",
				"url": "https://<REDACTED>",
				"header": {
					"Authorization": [ "Bearer <REDACTED>" ]
				}
			}
		}
	}
}

Config from Remote:

{
  "admin": {
    "listen": "<REDACTED>:2020",
    "identity": {
      "identifiers": [
        "<REDACTED>"
      ],
      "issuers": [
        {
          "module": "internal",
          "ca": "admin",
          "lifetime": "1d",
          "sign_with_root": false
        }
      ]
    },
    "remote": {
      "access_control": [
        {
          "public_keys": [
            "<REDACTED>"
          ]
        }
      ]
    }
  },
  "apps": {
    "http": {
      "servers": {
        "srv0": {
          "listen": [
            "<REDACTED>:443"
          ],
          "routes": [
            {
              "@id": "<REDACTED>",
              "handle": [
                {
                  "handler": "reverse_proxy",
                  "upstreams": [
                    {
                      "dial": "tcp\/<REDACTED>"
                    }
                  ]
                }
              ],
              "match": [
                {
                  "host": [
                    "<REDACTED>"
                  ]
                }
              ],
              "terminal": true
            }
          ]
        }
      }
    },
    "pki": {
      "certificate_authorities": {
        "local": {
          "install_trust": false
        },
        "admin": {
          "name": "Caddy",
          "root_common_name": "<REDACTED>",
          "intermediate_common_name": "<REDACTED>",
          "install_trust": false
        }
      }
    }
  }
}

3. The problem I’m having:

When making an API request with CURL to fetch the config, it returns the local config rather than the remote config.

curl --cert client.pem --key client-key.pem -k https://<REDACTED>:2021/config/ | jq (Port is 2021 due to mTLS)

Returns the exact same config specified in the local caddy.json, instead the remote or any merging of the two configs.

I’d expect that API request to return the remote config that was fetched.

4. Error messages and/or full log output:

❯ ./caddy run -config=caddy.json
2021/06/18 18:47:33.629 INFO    using provided configuration    {"config_file": "caddy.json", "config_adapter": ""}
2021/06/18 18:47:33.630 INFO    admin   admin endpoint started  {"address": "tcp/<REDACTED_PRIVATE_IP>:2020", "enforce_origin": false, "origins": ["<REDACTED_PRIVATE_IP>:2020"]}
2021/06/18 18:47:33.772 INFO    applying dynamically-loaded config      {"loader_module": "http"}
2021/06/18 18:47:33.772 INFO    autosaved config (load with --resume flag)      {"file": "/home/matthew/.config/caddy/autosave.json"}
2021/06/18 18:47:33.772 INFO    serving initial configuration
2021/06/18 18:47:33.772 INFO    admin   admin endpoint started  {"address": "tcp/<REDACTED_PRIVATE_DOMAIN>:2020", "enforce_origin": false, "origins": ["<REDACTED_PRIVATE_DOMAIN>:2020"]}
2021/06/18 18:47:33.773 INFO    http    server is listening only on the HTTPS port but has no TLS connection policies; adding one to enable TLS {"server_name": "srv0", "https_port": 443}
2021/06/18 18:47:33.773 INFO    http    enabling automatic HTTP->HTTPS redirects        {"server_name": "srv0"}
2021/06/18 18:47:33.773 INFO    tls.cache.maintenance   started background certificate maintenance      {"cache": "0xc00033ea10"}
2021/06/18 18:47:33.773 WARN    pki.ca.local    root certificate trust store installation disabled; unconfigured clients may show warnings      {"path": "storage:pki/authorities/local/root.crt"}
2021/06/18 18:47:33.773 WARN    pki.ca.admin    root certificate trust store installation disabled; unconfigured clients may show warnings      {"path": "storage:pki/authorities/admin/root.crt"}
2021/06/18 18:47:33.773 INFO    pki     intermediate expires soon; renewing     {"ca": "local", "time_remaining": -2072927.773478879}
2021/06/18 18:47:33.773 INFO    pki     renewed intermediate    {"ca": "local", "new_expiration": "2021/06/25 18:47:33.000"}
2021/06/18 18:47:33.773 INFO    pki     intermediate expires soon; renewing     {"ca": "admin", "time_remaining": -2072927.773892499}
2021/06/18 18:47:33.774 INFO    pki     renewed intermediate    {"ca": "admin", "new_expiration": "2021/06/25 18:47:33.000"}
2021/06/18 18:47:33.774 INFO    tls     cleaning storage unit   {"description": "FileStorage:/home/matthew/.local/share/caddy"}
2021/06/18 18:47:33.774 INFO    http    enabling automatic TLS certificate management   {"domains": ["<REDACTED_PUBLIC_DOMAIN>"]}
2021/06/18 18:47:33 [INFO] Certificate certificates/admin/<REDACTED_PRIVATE_DOMAIN>/<REDACTED_PRIVATE_DOMAIN>.crt expired 719h48m47.774966656s ago; cleaning up
2021/06/18 18:47:33 [INFO] Deleting certificates/admin/<REDACTED_PRIVATE_DOMAIN>/<REDACTED_PRIVATE_DOMAIN>.crt because resource expired
2021/06/18 18:47:33 [INFO] Deleting certificates/admin/<REDACTED_PRIVATE_DOMAIN>/<REDACTED_PRIVATE_DOMAIN>.key because resource expired
2021/06/18 18:47:33 [INFO] Deleting certificates/admin/<REDACTED_PRIVATE_DOMAIN>/<REDACTED_PRIVATE_DOMAIN>.json because resource expired
2021/06/18 18:47:33 [INFO] Deleting certificates/admin/<REDACTED_PRIVATE_DOMAIN> because key is empty
2021/06/18 18:47:33.775 INFO    tls     finished cleaning storage units
2021/06/18 18:47:33.872 INFO    admin.identity.obtain   acquiring lock  {"identifier": "<REDACTED_PRIVATE_DOMAIN>"}
2021/06/18 18:47:33.872 INFO    admin.remote    secure admin remote control endpoint started    {"address": "tcp/:2021"}
2021/06/18 18:47:33.872 INFO    dynamically-loaded config applied successfully
2021/06/18 18:47:33.874 INFO    admin.identity.obtain   lock acquired   {"identifier": "<REDACTED_PRIVATE_DOMAIN>"}
2021/06/18 18:47:33.875 INFO    admin.identity.obtain   certificate obtained successfully       {"identifier": "<REDACTED_PRIVATE_DOMAIN>"}
2021/06/18 18:47:33.875 INFO    admin.identity.obtain   releasing lock  {"identifier": "<REDACTED_PRIVATE_DOMAIN>"}
2021/06/18 18:47:33.876 WARN    admin.identity  stapling OCSP   {"error": "no OCSP stapling for [<REDACTED_PRIVATE_DOMAIN>]: no OCSP server specified in certificate"}
2021/06/18 18:47:34.272 INFO    admin   stopped previous server {"address": "tcp/<REDACTED_PRIVATE_IP>:2020"}
2021/06/18 18:48:11.294 INFO    admin.api       received request        {"method": "GET", "host": "127.0.0.1:2020", "uri": "/config/", "remote_addr": "127.0.0.1:39468", "headers": {"Accept":["*/*"],"User-Agent":["curl/7.61.1"]}}
2021/06/18 18:48:11.294 ERROR   admin.api       request error   {"error": "host not allowed: 127.0.0.1:2020", "status_code": 403}
2021/06/18 18:50:10.840 INFO    admin.api       received request        {"method": "GET", "host": "<REDACTED_PRIVATE_DOMAIN>:2021", "uri": "/config/", "remote_addr": "127.0.0.1:55172", "headers": {"Accept":["*/*"],"User-Agent":["curl/7.61.1"]}, "secure": true, "verified_chains": 1}
2021/06/18 18:50:16.051 INFO    admin.api       received request        {"method": "GET", "host": "<REDACTED_PRIVATE_DOMAIN>:2021", "uri": "/config/", "remote_addr": "127.0.0.1:55180", "headers": {"Accept":["*/*"],"User-Agent":["curl/7.61.1"]}, "secure": true, "verified_chains": 1}

5. What I already tried:

Not much I can try, I’m pretty convinced this is either a bug or is intentional behavior. I attempted to look through the code for the remote config loading but had no idea where I would possibly start trying to debug or figure this out.

6. Links to relevant resources:

matt · June 18, 2021, 7:52pm

It’s been a while since I looked at the config loading code, or thought much about its design, but I’m pretty sure it’s intentional: the idea is that the config you feed to Caddy is what gets regurgitated when you do a GET on the API. Crucially, it’s also what gets persisted to the autosave file so that on restart, the same config can be resumed, rather than the one that was dynamically loaded. Because once you replace the “original” config you fed Caddy with one it dynamically loads, the dynamically-loaded config becomes the new “original” config, or static; and you lose the dynamic loading capability. That seemed like a bad idea to me.

But nobody was willing to test it during the (very long) beta period, so

matthewpi · June 18, 2021, 8:07pm

Yeah, I understand that side and that’s why I was unsure if this was a bug or not. Could it be an option to have the API return the dynamic config rather it be through a config option, header, or query param? My use-case for this feature is to have Caddy initially load a config that is generated by a service and that service will also send requests to add new hosts when needed, making sure that when Caddy is restarted it gets the full config if it missed any requests or got out of sync somehow (rather than just having Caddy persist the config).

matt · June 20, 2021, 12:45am

Hmm, I don’t know. I do think it’s confusing to export 2 different configs. IMO, Caddy isn’t really responsible for keeping track of dynamically-loaded configs. If it’s pulling a config, then it’s usually the job of the external config generator (i.e. the source of the config) to keep track of things.

Your design doesn’t have a clear “owner” of the config. Is it your system, which is the source of the initial config, or Caddy, when your system starts pushing config changes? I think it’s important that one member of a system definitively owns the config.

system · July 18, 2021, 7:13pm

This topic was automatically closed after 30 days. New replies are no longer allowed.