Can i drain sessions from an upstream server?

Help
1

1. The question I’m having:

Hi,
we want to run Blue/Green deployments with caddy as a reverse proxy.

My questions:

  • Can i set one or multiple upstream servers to “gracefully” drain sessions?
  • Can i access a metric in caddy to see how many sessions are still active to an upstream server?
  • Has anyone done this before and can share an example implementation?

Our upstream servers are session aware, so we would like to “drain” them before replacing one of our deployments (wait for current sessions to close/expire and don’t create new sessions on these servers).

The only option i found, from searching the web and this community, was to remove the servers from the reverse_proxy pool or flag them as unhealthy - but this would affect active sessions on the servers as well - which i want to avoid.

The other solution i found is the community post linked below, but i’m not sure if this was ever fully implemented…

Let me know if i can provide further details to clarify my question :slight_smile:

Best regards
Alcesh

3. Caddy version:

v2.6.4

d. My complete Caddy config:

poc.domain {
		reverse_proxy * {
                to blue01:9000 blue02:9000 green01:9000 green02:9000
        }
}

5. Links to relevant resources:

2

That depends on how you define a “session”. Is a session more than a connection?

3

If you use v2.7.0-beta.2, you could use the new lb policy fallback, plus the new weighted_round_robin policy, which would help with this. See reverse_proxy (Caddyfile directive) — Caddy Documentation.

The idea is you’d use cookie as your primary load balancing policy, and use weighted_round_robin as the secondary (instead of the default of random) which would allow you to skip the green server from being chosen for new connections, while the cookie would allow connections to get routed to the green server if it had already hit it before.

reverse_proxy blue01:9000 blue02:9000 green01:9000 green02:9000 {
	lb_policy cookie {
		fallback weighted_round_robin 1 1 0 0
	}
}

You’d have to update the weights and reload the config anytime you want to swap it over.

1 Like
4

I thought a session would be a bunch of connections to the same server (sticky sessions with cookies). Now that you asked that question, i will get in touch with the application developers to clarify this, since i don’t know for sure how they implemented this.

The answer from francislavoie is what i was looking for.
I will check if we can PoC the beta version or if our project has a policy to wait for the final release before testing.
This seems like the combination of cookie and weights i’ve been looking for.

Is there a way to check in caddy if it has provided a cookie to a user that has not yet expired?

This would be a wonderful addition for re-deploying a blue/green instance in a controlled way: drain the sessions, wait for the cookies to expire, re-deploy the instance.

Thanks so far! :slight_smile:

5

The cookie Caddy adds doesn’t expire (currently). If it can’t route the request to the same upstream as mapped by the cookie value, then a new one will be chosen by the fallback policy and a new cookie will be written with the value being the hash to that upstream.

We could (and probably should) add optional expiration for the cookies to allow them to re-choose a new upstream after some time. How long would you set the expiry to?

6

How long would you set the expiry to?

For the main applications i work with (ticketing tools, remote sessions, …) at our company i would go with ~3 hours, since i switch between working in the tool, on servers and attending meetings all the time…
The internal session timeout / logout of most of our web based apps is between 1 to 3 hours, sometimes more, depending on security and user needs.

In my opinion this very much depends on your use case…
If i have a rather active session, i would set it to 1 hour, if i have a rater inactive session, i would set it to 3 hours…

So i would tend to a 3 hour default for starters…

7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.