How to get The reverse_proxy directive to strip the path of the request before forwarding the request upstream

sflamm · July 10, 2022, 9:31am

1. Caddy version (`caddy version`):

v2

2. How I run Caddy:

caddy run

a. System environment:

windows server 2019

b. Command:

caddy_windows_amd64.exe run

c. Service/unit/compose file:

Paste full file contents here.
Make sure backticks stay on their own lines,
and the post looks nice in the preview pane.

d. My complete Caddyfile or JSON config:

Paste config here, replacing this text.
Use `caddy fmt` to make it readable.
DO NOT REDACT anything except credentials.
LEAVE DOMAIN NAMES INTACT.
Make sure the backticks stay on their own lines.

3. The problem I’m having:

trying to match on a path prefix like /version/a/b/c which will choose the appropriate reverse proxy but then forward the request without the prefix version… so the request will just be /a/b/c

inbound request:

http//:server.acme.com:portx/version/a/b/c — reverse proxy → 127.0.0.1:portY/a/b/c

cannot figure out the proper syntax for this…

the problem is described here too " The reverse_proxy directive]does not strip the path of the request before forwarding the request upstream:

github.com/caddyserver/caddy

Strip path for reverse proxied requests

opened 12:51PM - 15 Apr 20 UTC

closed 09:27PM - 26 May 20 UTC

zakcutner

feature discussion

#### Background The [`reverse_proxy` directive](https://caddyserver.com/docs/…caddyfile/directives/reverse_proxy) does not strip the path of the request before forwarding the request upstream. For example, in the following configuration a request to `example.com/foo/bar` will result in an upstream request of `upstream:80/foo/bar` *(rather than `upstream:80/bar`)*. ``` example.com { route /foo* { reverse_proxy upstream:80 } } ``` At least from my experience, I quite often want the path to not be included in the upstream request, as the upstream server is a microservice which is not aware that it is being reverse-proxied. A good example given in #2813 (where this issue originated) is using phpMyAdmin at the URL `/phpmyadmin`. The [`uri` directive](https://caddyserver.com/docs/caddyfile/directives/uri) can be used to achieve this and the configuration can be changed to something like this. ``` example.com { route /foo* { uri strip_prefix /foo reverse_proxy upstream:80 } } ``` #### Issue As this seems to be a relatively common pattern, I imagine configurations like the following will become quite commonplace. ``` example.com { route /api { route /v1* { uri strip_prefix /api/v1 reverse_proxy oldapi:80 } route /v2* { uri strip_prefix /api/v1 reverse_proxy newapi:80 } } route /docs* { uri strip_prefix /docs reverse_proxy docs:80 } reverse_proxy website:80 } ``` Here there is a lot of duplication of paths, i.e. `/docs` is written twice. I would like to avoid this to make the configuration more succinct and because it could very easily lead to errors being accidentally introduced. For instance, if I change `/docs` to be `/documentation` I may forget to update `uri strip_prefix /docs` to `uri strip_prefix /documents`. In a more complex config like this one you can see how there may be more subtle errors with the api/ path as there is nesting too. #### Ideas I suggested a few ideas in #2813 and @francislavoie made some good points, also mentioning that in a lot of cases a subdomain may be preferred. Ideas that I have come up with are... 1. Change the default behaviour of `reverse_proxy` to strip out the path. Then you could simply add it back in by changing `reverse_proxy upstream:80` to `reverse_proxy upstream:80/foo`. However, @francislavoie pointed out that this could be a confusing change from the current behaviour in Caddy (both v1 and v2) and may be unintuitive to users who expect the path to be preserved. 2. Add an option to the `reverse_proxy` directive to strip the path something like in the following example. However, this seems to go against the good principle that the `reverse_proxy` directive should not be involved in URL rewrites. ``` route /foo* { reverse_proxy upstream:80 { strip_path } } ``` 3. Introduce a new [placeholder](https://caddyserver.com/docs/conventions#placeholders) (for instance `{route}` perhaps) which would return the current path matched by Caddy. Then it could be used like so to remove the duplication. However, I'm not entirely sure that this expresses clearly what we are trying to achieve. ``` route /foo* { uri strip_prefix {route} reverse_proxy upstream:80 } ``` 4. Similar to ideas 2 and 3, create a new option in the `uri` directive that allows the currently matched route to be stripped from the path like in the following. It could also take an optional argument of a maximum number of path segments to strip. However, this increases the complexity of the `uri` directive with more options. ``` route /foo* { uri strip_route reverse_proxy upstream:80 } ``` I'd be really interested to find out what use cases exist for this and whether there are any other ideas how to do this 💡

I tried the following config but it just returns a blank page instead of the content:

http://host.acme.com {

handle_path /v1 {

	reverse_proxy /a 127.0.0.1:10002 {
		transport http {
		}
	header_up Host {host} # redundant
	header_up X-Real-IP {remote}
	header_up X-Forwarded-For {remote}  # redundant
	header_up X-Forwarded-Port {server_port} # redundant
	header_up X-Forwarded-Proto {scheme}
	}
}

}

4. Error messages and/or full log output:

5. What I already tried:

6. Links to relevant resources:

francislavoie · July 10, 2022, 10:34am

sflamm:

		transport http {
		}
	header_up Host {host} # redundant
	header_up X-Real-IP {remote}
	header_up X-Forwarded-For {remote}  # redundant
	header_up X-Forwarded-Port {server_port} # redundant
	header_up X-Forwarded-Proto {scheme}

Remove all this, it’s not useful. Caddy sets the appropriate headers automatically:

Use the handle_path directive which will strip the matched path prefix before handling.

Your mistake is that you must use *, because path matching is exact in Caddy.

handle_path /version* {
	reverse_proxy 127.0.0.1:10002
}

sflamm · July 10, 2022, 11:18am

Thanks for the speedy reply. I updated the configuration to:

http://host.acme.com {
			
	handle_path /version1* {
   
		reverse_proxy  127.0.0.1:10002 {
			
	}
}

Now it no longer has a 404… but the page is blank… looking at the page source it looks like it has never loaded the scripts:

<body>
  <div id="root"></div>
<script src="[/devui/static/js/bundle.js](http://host.acme.com:8008/devui/static/js/bundle.js)"></script><script src="[/devui/static/js/vendors~main.chunk.js](http://host.acme.com:8008/devui/static/js/vendors~main.chunk.js)"></script><script src="[/devui/static/js/main.chunk.js](http://host.acme.com:8008/devui/static/js/main.chunk.js)"></script></body>
</html>

if I remove the handle_path from the configuration (and therefore the URL is http://host.acme.com:8008/a/b/c instead of http://host.acme.com:8008/version1/a/b/c the application loads perfectly:

http://host.acme.com {
		reverse_proxy  127.0.0.1:10002 {
	
}

what is the difference? nothing has changed on the application end…

the topology for reference:

—> host.acme.com:8008/version1/a/b [firewall] → host.acme.com/version1/a/b:80 [caddy] → app on Port 10002
both caddy and the application are running on the same server

francislavoie · July 10, 2022, 11:36am

Make sure your { match all the closing }. You have an extra one now at the end of the reverse_proxy line.

That’s not a problem with Caddy. You need to configure your upstream app to be aware of the subpath you’re proxying to, since it constructs the HTML which contain the script tags.

Using handle_path, Caddy will only handle requests which have /version1* in the path. Other requests will go unhandled, which results in an empty 200 response. You either need to configure Caddy to handle those other requests with another handle block, or configure your upstream app to avoid needing to do the handle_path in Caddy.

sflamm · July 10, 2022, 4:32pm

Thanks for the information.

In this common scenrio, what is needed is for the application to be unaware of the proxy.

To do that:

match on the prefix /versionNumber
strip the prefix so that the request is proxied to the application as norml but port based on /versionNumber (e.g. versionNumber = 1000, reverse_proxy localhost:1000)
rewrite the root for ALL urls found in the responses to the client to put back the /versionNumber prefix

Will this configuration do this?

handle_path /10002 {
		reverse_proxy  127.0.0.1:10002 {
			handle_response  {
		            root    * /10002
                            rewrite * /10002/{path}		
                       }
               }
}

This does NOT appear to be rewriting any of the embedded URIs

Also - the value of the path that is stripped by handle_path - is there a variable that can be used to get its value (e.g. $1)?

francislavoie · July 10, 2022, 5:06pm

rewrite only rewrites the URI of the incoming request. It doesn’t manipulate responses. Manipulating the request during handle_response doesn’t do anything, because the request has already been used/copied to the proxy upstream.

I don’t understand your goal here. I think you’re approaching this incorrectly. What exactly is your goal, and why are you trying to do this?

Also, please run caddy fmt --overwrite Caddyfile to fix your config’s formatting. It’s messy and hard to read.

sflamm · July 10, 2022, 5:22pm

the goal:

have multiple “versions” of an application running… where each version is based on the build number

each version will be running on a port corresponding to the build number: e.g. build 10000 running on port 10000

to access the application using same URI request that the application would normally expect but modified to add/embed the the version number into the path to do the correct reverse_proxy

  normal URL:      host.acme.com:originalPort/originalPath  -->  **127.0.0.1:originalPort**/originalPath
         modified URL:         host.acme.com:originalPort/**versionNumber**/originalPath --> **127.0.0.1:versionNumber**/originalPath

where the application is UNAWARE of any of the path manipulation

Can you let me know what configuration I can use to achieve this?

after running caddy fmt the configuration file currently is:

http://host.acme.com {
	handle_path /data* {
		reverse_proxy 127.0.0.1:10002 {
			handle_response {
				root * /10002
			}
		}
	}
}

francislavoie · July 10, 2022, 5:33pm

Why does it have to be unaware? That makes it significantly harder to do this.

If you can just have a base path parameter (environment variable maybe) for that deployment, or even make the base path dynamic based on some HTTP header during that app’s handling (that you could pass with header_up in Caddy, like X-Base-Path: /10000), then you wouldn’t have an issue.

sflamm · July 10, 2022, 5:35pm

Unfortunately we do not control the source for the application… need to test different versions independently…

This seems to be a very common need/scenario… where a component/prefix of the path is stripped to do the proxy to a specific port… and the response should be rewritten to include the stripped prefix/component on all URLs in the response

francislavoie · July 10, 2022, 5:38pm

It’s not, actually. Most upstream apps will have some config like basePath to control how it builds URLs.

This isn’t something Caddy can magically fix. As the article I linked above explains, you’d need to manipulate the response body, which is non-trivial. You’d have to use regular expressions to update any URLs in the content, but that’s very error-prone, because there’s no fool-proof way to detect URLs in the response body. If you try to replace anything that starts with / or whatever, then you’ll replace too many things, etc. Also, response body replacement is not built-in, you’d need to install a plugin to do it. And I strongly recommend against taking this approach, you’ll cause yourself more headaches.

sflamm · July 10, 2022, 5:45pm

appreciate your patience.

… will chat with the develpers to set the basepath based on the portnumbr that is built…

system · August 9, 2022, 9:31am

This topic was automatically closed after 30 days. New replies are no longer allowed.