ID Selector is not working after upgrade to 2.4.0

rehanvdm · May 19, 2021, 1:09pm

1. Caddy version (`caddy version`):

2.4.0

2. How I run Caddy:

a. System environment:

Reproduced on both Windows 10 and Amazon Linux 2.

b. Command:

For windows

caddy_win.exe run --resume

For Linux

sudo systemctl start caddy

c. Service/unit/compose file:

d. My complete Caddyfile or JSON config:

Just going to show the windows configuration from here on out.

{
    "admin": {
        "listen": ":2020"
    },
    "apps": {
        "http": {
            "servers": {
                "server443": {
                    "@id": "routes - global",
                    "listen": [
                        ":443"
                    ],
                    "routes": [
                        {
                            "handle": [
                                {
                                    "@id": "routes - dev.uafricastore.local",
                                    "handler": "subroute",
                                    "routes": []
                                },
                                {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "From": [
                                                    "localhost"
                                                ],
                                                "Host": [
                                                    "{http.request.host}"
                                                ],
                                                "X-Forwarded-Proto": [
                                                    "https"
                                                ]
                                            }
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "localhost:10047"
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "*.dev.uafricastore.local"
                                    ]
                                }
                            ]
                        },
                        {
                            "@id": "dev.uafricastore.local",
                            "handle": [
                                {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "From": [
                                                    "localhost"
                                                ],
                                                "Host": [
                                                    "{http.request.host}"
                                                ],
                                                "X-Forwarded-Proto": [
                                                    "https"
                                                ]
                                            }
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "localhost:10047"
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "dev.uafricastore.local"
                                    ]
                                }
                            ]
                        },
                        {
                            "handle": [
                                {
                                    "@id": "routes - uafricastore.local",
                                    "handler": "subroute",
                                    "routes": [
                                        {
                                            "@id": "42 - custom-redirect",
                                            "handle": [
                                                {
                                                    "handler": "static_response",
                                                    "headers": {
                                                        "Location": [
                                                            "https://testcustagain1.local"
                                                        ]
                                                    },
                                                    "status_code": 301
                                                }
                                            ],
                                            "match": [
                                                {
                                                    "host": [
                                                        "testcustagain1.uafricastore.local"
                                                    ]
                                                }
                                            ]
                                        },
                                        {
                                            "@id": "53 - custom-redirect",
                                            "handle": [
                                                {
                                                    "handler": "static_response",
                                                    "headers": {
                                                        "Location": [
                                                            "https://testchange2.local"
                                                        ]
                                                    },
                                                    "status_code": 301
                                                }
                                            ],
                                            "match": [
                                                {
                                                    "host": [
                                                        "testchange2.uafricastore.local"
                                                    ]
                                                }
                                            ]
                                        }
                                    ]
                                },
                                {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "From": [
                                                    "localhost"
                                                ],
                                                "Host": [
                                                    "{http.request.host}"
                                                ],
                                                "X-Forwarded-Proto": [
                                                    "https"
                                                ]
                                            }
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "localhost:10041"
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "*.uafricastore.local"
                                    ]
                                }
                            ]
                        },
                        {
                            "@id": "uafricastore.local",
                            "handle": [
                                {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "From": [
                                                    "localhost"
                                                ],
                                                "Host": [
                                                    "{http.request.host}"
                                                ],
                                                "X-Forwarded-Proto": [
                                                    "https"
                                                ]
                                            }
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "localhost:10041"
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "uafricastore.local"
                                    ]
                                }
                            ]
                        },
                        {
                            "@id": "19 - custom",
                            "handle": [
                                {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "Host": [
                                                    "{http.request.host}"
                                                ],
                                                "X-Forwarded-Proto": [
                                                    "https"
                                                ]
                                            }
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "127.0.0.1:10041"
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "custom4store.local",
                                        "www.custom4store.local"
                                    ]
                                }
                            ]
                        },
                        {
                            "@id": "27 - redirect",
                            "handle": [
                                {
                                    "handler": "static_response",
                                    "headers": {
                                        "Location": [
                                            "https://test9.uafricastore.local"
                                        ]
                                    },
                                    "status_code": 301
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "www.test9.local"
                                    ]
                                }
                            ]
                        },
                        {
                            "@id": "42 - custom",
                            "handle": [
                                {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "Host": [
                                                    "{http.request.host}"
                                                ],
                                                "X-Forwarded-Proto": [
                                                    "https"
                                                ]
                                            }
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "127.0.0.1:10041"
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "testcustagain1.local"
                                    ]
                                }
                            ]
                        },
                        {
                            "@id": "53 - custom",
                            "handle": [
                                {
                                    "handler": "reverse_proxy",
                                    "headers": {
                                        "request": {
                                            "set": {
                                                "Host": [
                                                    "{http.request.host}"
                                                ],
                                                "X-Forwarded-Proto": [
                                                    "https"
                                                ]
                                            }
                                        }
                                    },
                                    "upstreams": [
                                        {
                                            "dial": "127.0.0.1:10041"
                                        }
                                    ]
                                }
                            ],
                            "match": [
                                {
                                    "host": [
                                        "testchange2.local"
                                    ]
                                }
                            ]
                        }
                    ]
                }
            }
        }
    },
    "logging": {
        "logs": {
            "": {
                "level": "debug"
            }
        }
    }
}

3. The problem I’m having:

I can not use the @id selector to select the correct “section”. It returns a 200 status code but with no content.

This used to work on the previous version: v2.3.0 but stopped after upgrading v.2.4.0

I have tried on both Windows 10 and Ubuntu (running on AWS, AMI: ubuntu/images/hvm-ssd/ubuntu-focal-20.04-amd64-server-20201026)

4. Error messages and/or full log output:

curl --location --request GET 'http://localhost:2020/id/routes - global/routes/' \
--header 'Content-Type: application/json' \
--header 'Accept: application/json'

GET /id/routes%20-%20global/routes/ HTTP/1.1
Content-Type: application/json
Accept: application/json
User-Agent: PostmanRuntime/7.28.0
Cache-Control: no-cache
Postman-Token: d1d9532c-cf3b-45d9-8632-fd2ff13238d2
Host: localhost:2020
Accept-Encoding: gzip, deflate, br
Connection: keep-alive
 
HTTP/1.1 200 OK
Date: Wed, 19 May 2021 12:55:06 GMT
Content-Length: 0

5. What I already tried:

Escaping the spaces
Removing the - between the words
Replacing - with an underscore
Adding and removing the Accept and Content-Type headers
Different operating systems
Selecting using “direct ancestors” aka http://127.0.0.1:2020/config/apps/http/servers/server443/routes works

6. Links to relevant resources:

francislavoie · May 19, 2021, 2:55pm

What if you remove the spaces? Seems strange to me to be using non-URL-safe characters here (i.e. spaces; dashes are fine)

rehanvdm · May 19, 2021, 4:10pm

I have tried just a basic word and the one in the docs my_proxy nothing works. It definitely “finds” the ID because if I use an ID that does not exist, it gives an error complaining it can not be found. So I guess it finds the selector but it returns 0 data for it, Content-Length: 0

francislavoie · May 19, 2021, 4:12pm

Okay.

It’s most likely a regression introduced in this PR (cc @matt):

github.com/caddyserver/caddy

admin: Secure socket for remote management

caddyserver:master ← caddyserver:remote-admin

opened 10:32PM - 25 Jan 21 UTC

mholt

+862 -214

This PR adds 3 separate, but very related features: 1. Automated server ident…ity management 2. Remote administration over secure connection 3. Dyanmic config loading at startup ## 1. Automated server identity management How do you know you're connecting to the server you think you are? How do you know the server connecting to you is the server instance you think it is? Mutually-authenticated TLS (mTLS) answers both of these questions. Using TLS to authenticate requires a public/private key pair (and the peer must trust the certificate you present to it). Fortunately, Caddy is really good at managing certificates by now. We tap into that power to make it possible for Caddy to obtain and renew its own identity credentials, or in other words, a certificate that can be used for both server verification when clients connect to it, and client verification when it connects to other servers. Its associated private key is essentially its identity, and TLS takes care of possession proofs. This configuration is simply a list of identifiers and an optional list of custom certificate issuers. Identifiers are things like IP addresses or DNS names that can be used to access the Caddy instance. The default issuers are ZeroSSL and Let's Encrypt, but these are public CAs, so they won't issue certs for private identifiers. Caddy will simply manage credentials for these, which other parts of Caddy can use, for example: remote administration or dynamic config loading (described below). A bare-bones config might look like this: ```json { "admin": { "identity": { "identifiers": [ "123.123.123.123", "example.com", "127.0.0.1", "localhost" ], "issuers": [ { "module": "acme", "ca": "https://my-acme-server.example.com/", "trusted_roots_pem_files": ["my-acme-root.crt"] } ] } } } ``` Here, Caddy is told that its identities are those IP addresses and DNS names. It then will use your custom ACME server with a custom root certificate (to trust when connecting to it) to get certificates for those identifiers. Note that in this case, your CA would have to issue certs for localhost and 127.0.0.1, which most CAs don't do, since they can't be verified if they are remote. ## 2. Remote administration over secure connection This feature adds generic remote admin functionality that is safe to expose on a public interface. - The "remote" (or "secure") endpoint is optional. It does not affect the standard/local/plaintext endpoint. - It's the same as the [API endpoint on localhost:2019](https://caddyserver.com/docs/api), but over TLS. - TLS cannot be disabled on this endpoint. - TLS mutual auth is required, and cannot be disabled. - The server's certificate _must_ be obtained and renewed via automated means, such as ACME. It cannot be manually loaded. - The TLS server takes care of verifying the client. - The admin handler takes care of application-layer permissions (methods and paths that each client is allowed to use).\ - Sensible defaults are still WIP. - Config fields subject to change/renaming. Here's a basic example config, that I will explain: ```json { "admin": { "identity": { "identifiers": ["example.com"] }, "remote": { "access_control": [ { "public_keys": ["base64-encoded DER certificate"] } ] } } } ``` Explanation: - First we configure identity management. We tell Caddy that its identifier is `example.com`, so it will try to obtain and renew a certificate for that domain. By default, it will use publicly-trusted CAs. This is OK for DNS names that are properly configured. _Identity management is required when enabling remote administration, otherwise the server cannot present a TLS certificate to the client and secure the connection._ - We've enabled a secure admin endpoint at its default address **`:2021`** (you can customize it with `"listen": "..."` just like the regular admin endpoint) - note that the default address is not bound to a local interface, so it can be accessed remotely. - A single public key is then added to the ACL. Only the sole bearer of the associated private key is allowed unrestricted access of the API. We can also restrict different clients/users as for methods and paths they are allowed to access: ```json { "public_keys": ["base64-encoded DER certificate"], "permissions": [{ "paths": ["/id/foo/"], "methods": ["GET"] }] } ``` All the users specified in `public_keys` will be allowed to access all paths in the API starting with `/id/foo/` using only the `GET` method. As you can see, you can specify multiple paths and methods, and multiple groups of them, per group of public keys. Other advanced functionality is a bit limited because we cannot import any Caddy modules: they all import this package instead! So, we cannot import the `caddyhttp` or `caddytls` packages and take advantage of their advanced routing or security logic. The admin controls are relatively simple, but I imagine this should be more than enough...? Caddyfile config can probably be added later. ## 3. Dynamic config loading at startup Since this feature was planned in tandem with remote admin, and depends on its changes, I am combining them into one PR. Dynamic config loading is where you tell Caddy how to load its config, and then it loads and runs that. First, it will load the config you give it (and persist that so it can be optionally resumed later). Then, it will try pulling its _actual_ config using the module you've specified (dynamically loaded configs are _not_ persisted to storage, since resuming them doesn't make sense). This PR comes with a standard config loader module called `caddy.config_loaders.http`. Here's how it looks: ```json { "admin": { "config": { "load": { "module": "http", "url": "https://example.com/my_caddy_config.json" } } } } ``` Caddy will download the config at the given URL and run it. You can also configure authentication -- both client and server -- to ensure you get only trusted configs. If you add this to your config: ``` "tls": { "use_server_identity": true } ``` then Caddy will use the configured identity (explained above) as a client certificate to present to the server it is connecting to. In this case, identity management must also be configured.

matt · May 19, 2021, 4:18pm

That would be weird, those changes have nothing to do with the actual admin handlers, or config IDs… looking into it.

matt · May 19, 2021, 4:28pm

Looks like I touched an adjacent function and made a mistake when rebasing. Fixed on master now.

rehanvdm · May 19, 2021, 4:35pm

That was super fast thanks for the help…

I have only been downloading the Caddy executable from the website because I need to include the Route53 package.

How can I test this change you made, I have seen GitHub - caddyserver/xcaddy: Build Caddy with plugins being mentioned a few places. Do I just point it to the master branch/to that new commit of yours and then instruct it that I also want the Route53 package with it?

francislavoie · May 19, 2021, 4:41pm

Yep.

xcaddy build master --with https://github.com/caddy-dns/route53

rehanvdm · May 19, 2021, 4:43pm

Awesome thanks, I will try it tomorrow (+12hours) and let know if it worked.

rehanvdm · May 20, 2021, 12:36pm

@matt Thanks that worked… Guess I was the first one that upgraded and used that on the latest version then, would have thought it be more popular…

What do others do then, just rewrite the whole config as opposed to what I am doing now, which is changing specific routes? OR I guess if you simple logic then you can use the path based, just gets tricky when you have an array and when the ordering of routes matter.

Anyway, thanks a million!

francislavoie · May 20, 2021, 4:03pm

A large majority of users just use the Caddyfile and don’t make use of the API. And I would postulate that those that do typically want more stability over updating early.

FYI v2.4.1 is out as of last night, which includes this fix.

basil · May 20, 2021, 4:36pm

I just tried using caddy upgrade, but the version is still 2.4.0??

francislavoie · May 20, 2021, 4:45pm

The build server’s build cache still hasn’t been kicked I think. See the explanation here:

system · June 18, 2021, 1:09pm

This topic was automatically closed after 30 days. New replies are no longer allowed.