Reverse proxy & rewriting

pdhar · February 20, 2018, 6:29pm

Hi , I am having some difficulty to make multiple reverse proxies work. I have a number of micro services running in docker-compose and reachable via their service name by custom networking(bridge).

/ is served by service http://webserver:3000
/login is served via http://loginservice:4300
/wistudio served via http://wistudio:7747

Each microservice needs the token which is the cookie returned from the /login service
For the first time the request to the CADDY_PROXY_PORT should check if the Token is set, if not rewrite to /login proxy and the proxy with strip /login and route to the service.

This works and the login upstream service redirects back to /applications which is then handled by /web_server rewrite to proxy /web_server.

The html returned by web_server has scripts which load a URI /wistudio/dist/file.json which goes through the rewrite for uri matching ^/wistudio/.*$ to proxy /wi_studio{uri}. This where it ends and I dont see the proxy /wi_studio take it upstream as per this log as the {upstream} placeholder is empty.

172.18.0.1 - - [20/Feb/2018:18:23:41 +0000] "GET /wistudio/dist/moduleConfig.json HTTP/1.1" 404 157 "http://localhost:8090/applications" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_13_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167 Safari/537.36"  UPSTREAM: - REWRITE: /wi_studio/wistudio/dist/moduleConfig.json TOKEN: - COOKIE:eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJjYyI6Ii0xIiwiaWNtb2NrIjoiMjAiLCJpY3dpIjoiMjAiLCJpY2J3IjoiMjAiLCJpY25vZGUiOiIyMCIsInVuYW1lIjoiamFjayIsInNic2MiOiJqYWNrX3Nic2MiLCJhY2N0cyI6WyJqYWNrX2FjY3QiXSwiZm4iOiJqYWNrLmZpcnN0IiwibG4iOiJqYWNrLmxhc3QiLCJleHAiOjE1MTkyMzc0MjAsImlhdCI6MTUxOTE1MTAyMCwibmJmIjoxNTE5MTUxMDE5LCJpc3MiOiJUQ0NfSWRNIiwiYXVkIjpbIlRDSSJdLCJldWxYIjp0cnVlLCJldWxhIjp0cnVlLCJyZWciOnRydWUsImVtYWlsIjoic2hvdWxkLm5vdGJlQGV4

I am not sure what is preventing multiple upstream reverse proxy to work. Any suggestions is appreciated.

My Caddy file is :

:{$CADDY_PROXY_PORT} {
    gzip
    root {$DATA_DIR}/proxy
    log   / stdout  "{combined}  UPSTREAM: {upstream} REWRITE: {rewrite_uri} TOKEN: {>X-App-Token} COOKIE:{~AppSession}"


    # CORS allow all
    cors /

    # Re-write / to /login if X-App-Token does not exist
    rewrite {
        if {>X-Atpp-Token} is  ""
        if {path} is /
        to /login{uri}
    }
    # Rewrite the URI if path matches regex /wistudio/.*$
    rewrite {
        regexp ^/wistudio/.*$
        to /wi_studio{uri}
    }

    # Re-write / to /web_server
    rewrite {
        #regexp ^/.*$
        to  /web_server{uri}
    }

    
    proxy /login {$LOGIN_SERVICE} {
        # Set downstream header cookie
        header_downstream X-App-Token {~AppSession}
        header_upstream X-App-Token {~AppSession}
        without /login
    }

    # Proxy /wi_studio
    proxy /wi_studio {$WISTUDIO_SERVICE} {
        transparent
        without /wi_studio
        # Set upstream header cookie
        header_upstream X-App-Token {~ AppSession}
        header_downstream X-App-Token {~ AppSession}
    }

    # Proxy /web_server and remove proxy prefix since 
    # it is the / 
    proxy /web_server {$WEB_SERVICE} {
        #transparent
        without /web_server
        except /wistudio 
        # Set upstream+downstream header cookie
        header_upstream X-App-Token {~AppSession}
        header_downstream X-App-Token {~AppSession}
    }


}

Whitestrake · February 20, 2018, 11:32pm

This seems like an interesting one. If you directly request /wi_studio/foo/bar, does the proxy work?

Also, I don’t know if the space in {~ AppSession} would cause issues or not.

pdhar · February 21, 2018, 12:44am

Sorry for my typo on {~ AppSession} , i actually abbreviated it to post here.
The /wi_studio/foo/bar did not work, however I tried another workaround based on the post Proxy/Rewrite Priority. It allowed the /wistudio proxy to work.

my caddy file looks like this now: -

Blockquote

0.0.0.0:{$CADDY_PROXY_PORT} {
    gzip
    root {$DATA_DIR}/proxy
    log   / stdout  "{combined}  UPSTREAM: {upstream} REWRITE: {rewrite_uri} TOKEN: {>X-App-Token} COOKIE:{~AppSession}"


    # CORS allow all
    # cors /

    # Re-write / to /login if X-App-Token does not exist
    rewrite {
        if {>X-App-Token} is  ""
        if {path} is /
        to /login{uri}
    }

    rewrite {
        regexp ^/wistudio/.*$
        to /wi_studio{uri}
    }

    # Re-write / to /web_server
    rewrite {
        #regexp ^/.*$
        to  /web_server{uri}
    }

    
    proxy /login {$LOGIN_SERVICE} {
        # Set downstream header cookie
        header_downstream X-App-Token {~AppSession}
        header_upstream X-App-Token {~AppSession}
        without /login
    }

    

    # Proxy /web_server and remove proxy prefix since 
    # it is the / 
    proxy /web_server {$WEB_SERVICE} {
        #transparent
        without /web_server
        except /wistudio 
        # Set upstream+downstream header cookie
        header_upstream X-App-Token {~AppSession}
        header_downstream X-App-Token {~AppSession}
    }
  
}

# Proxy /wistudio
0.0.0.0:{$CADDY_PROXY_PORT}/wistudio {
    proxy / {$WISTUDIO_SERVICE} {
        transparent
        # without /wi_studio
        # Set upstream header cookie
        header_upstream X-App-Token {~AppSession}
        header_downstream X-App-Token {~AppSession}
    }
}

Whitestrake · February 21, 2018, 1:21am

I edited your post slightly to format the Caddyfile in a code block (using three backticks, ```).

It’s very interesting that the form

example.com {
  proxy /foo upstream
}

completely failed to send a matching request upstream while the form

example.com/foo {
  proxy / upstream
}

succeeded. But if it works, it works!

You just might want to make sure the 0.0.0.0:{$CADDY_PROXY_PORT}/wistudio site block has any relevant directives (gzip, log) and a valid site root is always a good idea for security.

pdhar · March 7, 2018, 6:06am

Thanks for your reply. I have another interesting problem. In the caddy configuration above that worked the micro services running are 2 angular2 application microservices behaving as a single page application. In angular there is a notion of routes which are handled in the browser by the angular framework and these routes are not served by the webserver and wistudio microservices. The bootstrapping of the angular application starts on the webserver which then loads pages from wistudio microservice using browser routes. Upto this point this is working as expected however when it comes to refresh of a page e.g. /wistudio/connections the wistudio service has no clue and therefore the request must be redirected to webserver. Therefore I created a rewrite rule in
0.0.0.0:{$CADDY_PROXY_PORT}/wistudio section as shown below

# Proxy /wistudio
0.0.0.0:{$CADDY_PROXY_PORT}/wistudio {
    gzip
    root {$DATA_DIR}/proxy
    log   / stdout  "{combined}  UPSTREAM: {upstream} REWRITE_WISTUDIO: {rewrite_uri} TOKEN: {>X-Atmosphere-Token} COOKIE:{~AppSession}"
    cors

    # Rewrite the URI if path matches regex /wistudio/.*$
    rewrite {
        regexp ^/(connections|flow|connectiondetails|create-flow|extensions).*$
        to /webserver{uri}
    }

    # Proxy /wi_studio
    proxy /webserver {$WEB_SERVICE} {
        transparent
        without /webserver
        # Set upstream header cookie
        header_upstream X-App-Token {~AppSession}
        header_downstream X-App-Token {~AppSession}
    }

    proxy / {$WISTUDIO_SERVICE} {
        transparent
        # without /wi_studio
        # Set upstream header cookie
        header_upstream X-App-Token {~AppSession}
        header_downstream X-App-Token {~AppSession}
    }
}

The rewrite rule catches the angular browser routes and sends them back to webserver. This also worked, however I was trying to find if there can be a 404 redirect back to webserver service if the route is not handled in wistudio service. I am not keen on listing the angular browser routes in the rewrite rule as they are application dependent and can change.

Whitestrake · March 7, 2018, 6:35am

So if I’m understanding this right, you might have a client connect to an Angular app through Caddy, then browse around these app routes entirely inside the front-end, and then when they refresh the page (forcing a new connection to the server for whatever resource they’re on at the time), Caddy might not know to send to the right back-end? Sounds like an interesting obstacle!

So between the webserver upstream and the wistudio upstream, you want Caddy to check wistudio, and if it receives a 404, proxy to the webserver instead?

I don’t believe Caddy can do that. Closest I can think of would be rewriting to a proxy if the static file server can’t find a resource on disk, which is pretty common.

pdhar · March 7, 2018, 7:27am

You understanding is on the mark. I tried with a / rewrite in the wistudio section and in the to I put the proxy to /wi_studio and then /webserver as a next available route and changed the proxy / => /wi_studio'. Couple of issues that popped up, the without =>/webserver’ needed to be /webserver/wistudio , in the end it did not work for the app.

But when I look back at the original issue of having all the proxies in the same section instead of virtual host sections, it did not work and your comment suggested that the behavior was not expected. The space in {~ AppSession} was a typographical mistake on my part while writing the post. Are there any precedence on proxy uri matches or it is similar to nginx where the longest match wins. Any clues on where I went wrong in the first place.

pdhar · March 7, 2018, 4:27pm

Although the rewrite directive allows you to have multiple end targets in the to option, which allows a failure in the first target to check the next. Can similar failure section be added to the proxy, although the proxy allows multiple to targets for load balancing but does not have a failure redirection ?

Whitestrake · March 7, 2018, 11:58pm

I wasn’t actually 100% sure about that, so I took a look. Well, turns out I thought you said “rewrite” rather than “proxy” and investigated that instead.

Here’s what I found for that:

Rewrite directive precedence

The handler for rewrite is here:

https://github.com/mholt/caddy/blob/5552dcbbc7f630ada7c7d030b37c2efdce750ace/caddyhttp/rewrite/rewrite.go#L48-L55

We can see that it makes use of a ConfigSelector from httpserver, which looks like it’s provided for this purpose. Taking a closer look at the Select function:

caddy/caddyhttp/httpserver/middleware.go at 5552dcbbc7f630ada7c7d030b37c2efdce750ace · caddyserver/caddy · GitHub

Indicates that it chooses the longest-length BasePath that matches. What’s the BasePath? It’s set here by the rewrite setup function:

caddy/caddyhttp/rewrite/setup.go at 5552dcbbc7f630ada7c7d030b37c2efdce750ace · caddyserver/caddy · GitHub

So / is used as a default, but it is overridden if you use a base path in your directive e.g. rewrite /foo/bar { ... results in a BasePath of /foo/bar.

That means that because the ConfigSelector iterates through the list of rewrite rules, only replacing the selected rule if the new rule is longer, if multiple catch-all rewrites are encountered it will use the first one added to the rule list (I believe that would be the first rewrite in the Caddyfile). Perhaps @matt could confirm if my findings are correct.

And it turns out it’s completely irrelevant with respect to proxy as it uses its own matcher:

github.com

caddyserver/caddy/blob/5552dcbbc7f630ada7c7d030b37c2efdce750ace/caddyhttp/proxy/proxy.go#L270-L285


      
          // match finds the best match for a proxy config based on r.
          func (p Proxy) match(r *http.Request) Upstream {
          	var u Upstream
          	var longestMatch int
          	for _, upstream := range p.Upstreams {
          		basePath := upstream.From()
          		if !httpserver.Path(r.URL.Path).Matches(basePath) || !upstream.AllowedPath(r.URL.Path) {
          			continue
          		}
          		if len(basePath) > longestMatch {
          			longestMatch = len(basePath)
          			u = upstream
          		}
          	}
          	return u
          }

Which indicates conclusively that it follows the same behaviour as rewrite anyway - taking the longest match, using the first found in cases of equal-length base paths.

The setup function for proxy calls this to parse the provided configuration, adding new configs to the end, so the first instance of the longest base path of a proxy in the Caddyfile wins.

github.com

caddyserver/caddy/blob/5552dcbbc7f630ada7c7d030b37c2efdce750ace/caddyhttp/proxy/upstream.go#L182


      
          				if strings.Contains(hostHeader, "{host}") {
          					upstream.HealthCheck.Host = strings.Replace(hostHeader, "{host}", host, -1)
          				}
          			}
          			upstream.wg.Add(1)
          			go func() {
          				defer upstream.wg.Done()
          				upstream.HealthCheckWorker(upstream.stop)
          			}()
          		}
          		upstreams = append(upstreams, upstream)
          	}
          	return upstreams, nil
          }
          
          func (u *staticUpstream) From() string {
          	return u.from
          }
          
          func (u *staticUpstream) NewHost(host string) (*UpstreamHost, error) {
          	if !strings.HasPrefix(host, "http") &&

I mean, this functionality could easily be achieved, but the question is how to do it in a way that is simple, intuitive, and works with the existing internal workings of Caddy, especially the Caddyfile.

I would pose the question as to whether you would be better served writing another microservice (a tiny Go program would suit quite well) to write your own custom request routing logic, wherein you can write your own try-fail-retry proxy policy to your upstreams, which would free massive amounts of complexity from your Caddyfile.

pdhar · March 8, 2018, 4:01pm

This is where I think my first caddy.conf did not work as expected, since there were multiple catch-all rules , but the rules did have different regex sub rule for the required action. I guess all catch-all rules needs to be evaluated for a rewrite result as some of them will not return one after the sub-rule is checked, and the one that returns wins the rewrite.

Whitestrake · March 9, 2018, 12:15am

For rewrites specifically, the config selector uses the rewrite code’s own Match function:

https://github.com/mholt/caddy/blob/5552dcbbc7f630ada7c7d030b37c2efdce750ace/caddyhttp/rewrite/rewrite.go#L147-L169

This, I understand, ensures that it will select the first rewrite among base paths of equal lengths where that rewrite satisfies its conditionals and regex checks.

So the first one that would actually return a result, essentially.

system · June 7, 2018, 12:15am

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.