V2: Subfolder proxy to upstream root

mandusm · February 24, 2020, 9:27pm

1. My Caddy version (`caddy version`):

v2.0.0-beta.14 h1:QX1hRMfTA5sel53o5SuON1ys50at6yuSAnPr56sLeK8=

2. How I run Caddy:

bin/caddy run -config ~/caddy/conf/default.caddy -adapter caddyfile

a. System environment:

Linux virtual-host-1 5.4.7-arch1-1 #1 SMP PREEMPT Tue, 31 Dec 2019 17:20:16 +0000 x86_64 GNU/Linux

Running as local binary downloaded from Github releases

d. My complete Caddyfile or JSON config:

{
    email <redacted>
    http_port  9080
	https_port 9443
    # experimental_http3
    acme_ca https://acme-staging-v02.api.letsencrypt.org/directory
}

:9080 {
    encode gzip
    reverse_proxy /* 127.0.0.1:8123 {
        header_up X-Forwarded-For {remote_host}
    }
}

:9080/portainer/* {
    strip_prefix portainer/
    reverse_proxy localhost:9000 {
        header_up Host {host}
        header_up X-Real-IP {remote_host}
        header_up X-Forwarded-For {remote_host}
    }
}

3. The problem I’m having:

I’m tryin to route requests from a subfolder path to the root of a service running on another port.
Something I have been doing on Nginx for a long time. See the end of this post for the current, working NGINX config I am trying to recreate.

The “error” is very ambiguous. Caddy is not throwing any error.
When I open the developer console, I can see that it gets the root response. I get a 200 response.
Anything else it tries to get like /portainer/vendor.js does not load, and I get no error in caddy2

4. Error messages and/or full log output:

2020/02/24 21:21:19.598 INFO    using provided configuration    {"config_file": "/home/<redact>/caddy/conf/default.caddy", "config_adapter": "caddyfile"}
2020/02/24 21:21:19.602 INFO    admin   admin endpoint started  {"address": "localhost:2019", "enforce_origin": false, "origins": ["localhost:2019"]}
2020/02/24 21:21:19.602 INFO    http    server is listening only on the HTTP port, so no automatic HTTPS will be applied to this server {"server_name": "srv0", "http_port": 9080}
2020/02/24 13:21:19 [INFO][cache:0xc0000d9360] Started certificate maintenance routine
2020/02/24 21:21:19.604 INFO    tls     cleaned up storage units
2020/02/24 21:21:19.605 INFO    autosaved config        {"file": "/home/<redact>/.config/caddy/autosave.json"}
2020/02/24 21:21:19.605 INFO    serving initial configuration

5. What I already tried:

Pretty much everything.
Been googling, looking in these forums, and testing all sorts of combinations of the directives and matcher configs in my caddyfile.
None of it seems to work and most guidance I have gotten on this kind of thing is for Caddy V1, and I don’t see a direct translation to Caddy2

6. Links to relevant resources:

server {
    listen 443 ssl;

    location / {
        proxy_pass http://172.17.0.1:8123;
        proxy_set_header Host $host;
        proxy_redirect http:// https://;
        proxy_http_version 1.1;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
    }

    location /portainer/ {
        rewrite ^/portainer(/.*)$ $1 break;
        proxy_pass  http://172.17.0.1:9000/;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection $connection_upgrade;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto https;
        proxy_redirect    off;
    }

}

francislavoie · February 24, 2020, 11:32pm

Could you be more specific? Do those files return a specific HTTP status code?

The only thing I can think of that might be incorrect is the strip_prefix directive. I think you want it to be strip_prefix /portainer so that it doesn’t strip the leading slash from the path. I’m not sure though.

Also, you can simplify reverse_proxy /* 127.0.0.1:8123, you can remove the /*.

mandusm · February 25, 2020, 2:12am

Changing the strip_prefix directive to one you have suggested won’t work.
Taken from the docs

is the prefix to strip from the request path. This value may omit the leading forward slash / and it will be assumed.

After a bunch more tinkering… This seems to have solved it… in a way…

:9080/portainer* {
    uri_replace * /portainer/ /
    reverse_proxy localhost:9000 {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
    }
}

This still requires me to add the trailing / to my URI though… Which I don’t really like…

matt · February 25, 2020, 2:19am

I think what @francislavoie is looking for (and I agree would be helpful) is a specific curl -v command (and its output!) that demonstrates the bug / “not working” part. Because strip_prefix should do what you’d expect. Also, replacing reverse_proxy with respond {uri} may be helpful in debugging rewrites with curl. We’d like to see these bits so we can assess whether there’s a problem in Caddy or in your config.

mandusm · February 25, 2020, 5:59am

Thanks, I understand the request for more info and logs, but to be honest, if I had to post every config I have tried as well as the DEBUG output and the curl headers or chrome debug logs this would be a very messy post to decipher. So I have kept it to only the things that have brought me close to the solution.

I got it to work with the following CaddyFile config. I left some commented stuff in there so you can see what I have tried before.

:9080 {
    encode gzip
    # respond /portainer "Hello Portainer"
    redir /portainer /portainer/

    reverse_proxy /* 127.0.0.1:8123 {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
    }

}


:9080/portainer/* {
    # respond /portainer "Hello Portainer"
    # uri_replace * /portainer/ /
    strip_prefix portainer
    reverse_proxy localhost:9000 {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
    }
}

I’m not sure if I am just misunderstanding the way that matchers work or if the documentation just isn’t clear, but even though this works, I don’t feel like this is exactly the expected config to achieve what I wanted to achieve.

Things I learned from this so far.

The Caddy V2 docs still need a lot of work, the examples are misleading and it doesn’t make clear where options, variables etc are available for which directives.
Every subfolder I want to proxy needs to be defined as a new site block where I can do strip_prefix on. Trying to reverse_proxy a request to a backend inside the root site_block doesn’t work because it strips the prefix before the proxy processes it. A little annoying, but this will just force me to use snippets for common things like compression etc.
There are other ways of getting this to work too.

:9080 {
    encode gzip
    # respond /portainer "Hello Portainer"
    # redir /portainer /portainer/

    reverse_proxy /* 127.0.0.1:8123 {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
    }

}


:9080/portainer* {
    # respond /portainer "Hello Portainer"
    redir /portainer /portainer/
    #uri_replace * /portainer/ /
    strip_prefix portainer
    reverse_proxy localhost:9000 {
        header_up Host {http.request.host}
        header_up X-Real-IP {http.request.remote}
        header_up X-Forwarded-For {http.request.remote}
        header_up X-Forwarded-Port {http.request.port}
        header_up X-Forwarded-Proto {http.request.scheme}
    }
}

3.a. The matching in the siteblock makes sense. I understand that it now grabs everything with portainer and that’s why the redir directive now moves into the portainer block.
3.b. Why I need to do a redir of the portainer without the trailing slash, I don’t understand.

Whitestrake · February 25, 2020, 6:07am

Can you show us specifically which examples misled you, and which directives did not show its options/variables/etc comprehensively? I know I’d love to see the docs improved, and knowing which ones need attention will help us get PRs happening to update the docs (which are located here: GitHub - caddyserver/website: The Caddy website).

You can handle things as a group by using route to batch a series of directives in order on it.

Like:

route /api/* {
  strip_prefix /api
  reverse_proxy 127.0.0.1
}

True! Caddy v2 is quite flexible.

At a guess, this would be Portainer’s issue. Caddy passes on the request faithfully. If Portainer doesn’t like it if you make the request without a trailing slash, Caddy will return Portainer’s response to the client regardless. I note your nginx config has this same handling present, but as a transparent internal rewrite instead.

We don’t mind messy. The code blocks have scrolling sections, so posting the lot isn’t going to spam out the page. We know what to look for.

Also, we don’t need literally every permutation - just your latest, best effort attempt, along with some example requests and their responses (via curl is fantastic because it gives us a comprehensive, accurate representation of the mechanics of the request).

Anyway, I gave your initial config a shot as an exercise, based on the nginx configuration. Here’s what I came up with:

:9080 {
  encode gzip

  # Portainer
  redir /portainer /portainer/
  route /portainer/* {
    strip_prefix /portainer
    reverse_proxy 172.17.0.1:9000 {
      header_up Host {host}
      header_up X-Real-IP {remote_host}
      header_up X-Forwarded-For {remote_host}
      header_up X-Forwarded-Proto {scheme}
    }
  }

  # Main service
  reverse_proxy 172.17.0.1:8123 {
    header_up X-Forwarded-For {remote_host}
  }
}

mandusm · February 25, 2020, 6:24am

Oh Nice!
Thanks @Whitestrake, I was not aware of the route directive.

I gave that a swing with this config and it seems to work.

:9080 {
    encode gzip
    # respond /portainer "Hello Portainer"
    # redir /portainer /portainer/

    redir /portainer /portainer/
    route /portainer/* {
        strip_prefix portainer
        reverse_proxy localhost:9000 {
            header_up Host {http.request.host}
            header_up X-Real-IP {http.request.remote}
            header_up X-Forwarded-For {http.request.remote}
            header_up X-Forwarded-Port {http.request.port}
            header_up X-Forwarded-Proto {http.request.scheme}
        }
    }

    route /* {
        reverse_proxy 127.0.0.1:8123 {
            header_up Host {http.request.host}
            header_up X-Real-IP {http.request.remote}
            header_up X-Forwarded-For {http.request.remote}
            header_up X-Forwarded-Port {http.request.port}
            header_up X-Forwarded-Proto {http.request.scheme}
        }
    }
}

Just a quick cursory read of the docs for route seems to indicate that order of config is important. So, I have to define the least specific rules, last. Like the overall, main service redirect.
Which would make sense.

I did a quick test with some other upstream services I have and they all act the same here. I need to redir from the non slashed to the slashed. It’s a minor thing, just not sure why it’s happening.

Just saw this after I also tinkered. Thanks!

Whitestrake · February 25, 2020, 6:28am

I probably wouldn’t bother using route /*. All you’re doing here is adding another few lines to your Caddyfile and manually overriding the order of directives inside it. It’s going to catch everything otherwise.

Yeah, not every upstream is going to give you the same behaviour, but a lot will.

It’s even different between the Caddy v1 and the Caddy v2 docs! Just look at this:

https://caddyserver.com/v1/docs/ - doesn’t work!
https://caddyserver.com/v1/docs - works fine!

Welcome — Caddy Documentation - works fine!
Welcome — Caddy Documentation - redirects to add trailing slash!

Although in the Caddy website’s case, this is based on a difference in how the docs are laid out (I believe the v1 docs is a file called docs.md or similar in the web root, whereas v2, we’re talking about an index.html file in a /docs/ folder).

Yep, trailing slashes are annoying. I’d much prefer an opinionated upstream generate its own redirects to gracefully handle mistakes.

Speaking of docs - please don’t pass up the opportunity to point out specific places where we can improve our docs. Even if you can’t tell us how we should improve them - although suggestions are very welcome - just pointing us in the right direction is helpful.

matt · February 25, 2020, 6:35am

Seconding what @whitestake said: definitely feel free to discuss specific improvements we can make to our docs here: GitHub - caddyserver/website: The Caddy website – it’s been a bit of scramble getting everything as far as it has come to this point, so fast! (It’s been less than a year, from scratch.)

Ah yes, so, the old website (v1) and the new website (not v1) are constructed differently, for one thing; in addition, some trailing slash redirects are manually configured because the JS on the page needs them. The Caddy 2 docs, as of right now, actually consist of two SPAs, so we have to manually redirect for the trailing slash in some cases. In the general cases for normal static files, though, the file server will canonicalize the URI by adding or removing trailing slashes using HTTP redirects. This is optional in Caddy 2 and can be disabled if you don’t want that. Otherwise, all URIs are treated literally in Caddy 2 AFAIK.

When reverse proxying, many backends are (too) picky about trailing slashes on URIs, so you’ll have to accommodate them in your web server/proxy config (or fix the backend), oh well.

PS. I also second @whitestrake’s suggestion to not use route /* as you are in your config – it may work, but it’s kind of unnecessary I think. If you prefer how nginx configs are structured with location blocks instead, you may find the handle directive convenient in Caddyfile, but otherwise, I think you can just simply remove that second route block.

Whitestrake · February 25, 2020, 6:41am

Technically you could use handle for both!

Since reordering technically isn’t required between just strip_prefix and reverse_proxy (they’re already ordered this way by default: Caddyfile Directives — Caddy Documentation). Route is good because you know it’ll go off in the order you put it in. handle doesn’t mess with the default order.

Semantics, though. Just another way v2 is pretty flexible.

matt · February 25, 2020, 3:36pm

That’s all true, but since the ordering was already correct with strip_prefix, I’m still not sure why that didn’t work.

Whitestrake · February 25, 2020, 11:23pm

The issue was that it wasn’t batched.

The prefix would be stripped and then it would be ineligible for the proxy, or there would be no way to differentiate it from the base proxy.

Basically either handle or route was needed to ensure the prefix was stripped and it was immediately proxied after.

matt · February 26, 2020, 2:26am

Ah, now I see. Thanks, that makes sense.

system · May 26, 2020, 2:26am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.