Newbie setting up 2 upstream servers. healthcheck strategy Q

Help
1

1. Caddy version:

2.6.2

2. How I installed, and run Caddy:

a. System environment:

Ubuntu 22.04.1 LTS (jammy) on Upcloud

b. Command:

caddy run

d. My complete Caddy config:

hopkins.aaa.com {
reverse_proxy * {
to http://aaa.50.59.1 http://aaa.44.33.2
health_uri /logon.html
}
}

3. The problem I’m having:

I have a question about a “failover” server.
I use Tomcat and an application that requires user login.
We are trying to create a 2nd server so if/when the 1st one goes down, traffic goes to the 2nd server automatically. This requires another login to the app by all endusers (which CAN be annoying, but its better to have them back on immediately that wait for a support person to get the server back up).
So imagine being a user on the system and having to login because it transitioned to the backup server.
Now a support person fixes the issue on the 1st server and WHAT WE DO NOT WANT is all the users to have to login AGAIN when the “health” is restored and caddy starts routing back to server 1 (that would be even MORE annoying).

So I’m unsure how to setup a system where its not so much PRIMARY and SECONDARY, but something more like CURRENT and BACKUP. And then when the health of the CURRENT is bad, the BACKUP starts getting traffic and IT becomes the CURRENT.
I’d like to correct the issue on the down server and get it back online, but don’t want traffic to go back to it UNLESS the health of the NOW CURRENT server goes bad, and on and on.

I’m wondering if the API is the only way to do this.

Also, is there something built into caddy where a shell command can be run when the health of PRIMARY goes bad? If caddy is going to determine that my traffic needs to go to the SECONDARY server, I’d like to send an email to someone.

Thx
Jaymer…

2

See docs/caddyfile/directives/reverse_proxy#load-balancing

See docs/api#get-reverse-proxyupstreams or the - somewhat experimental and not too well documented - event hooks using GitHub - mholt/caddy-events-exec: Run commands on Caddy events

3

It depends what you mean by “login again”. How is the authentication done? For example, If you’re using cookie-based auth, then the browser will send the cookie on any request to your server, and it’s up to your server to look up the session based on the auth cookie value in your cache or DB. So there’s no concern there at all.

5

thanks, but that doesn’t apply to me. Not how Tomcat handles sessions.

“So I’m unsure how to setup a system where its not so much PRIMARY and SECONDARY, but something more like CURRENT and BACKUP. And then when the health of the CURRENT is bad, the BACKUP starts getting traffic and IT becomes the CURRENT.
I’d like to correct the issue on the down server and get it back online, but don’t want traffic to go back to it UNLESS the health of the NOW CURRENT server goes bad, and on and on.”

6

Was there something SPECIFIC in that link to the docs you wanted me to see?

These docs are highly technical, with little examples, and certainly not for my case.
BTW, I’ve seen the docs.
But I’d appreciate your input on how to solve my issue.

w.r.t. to the events, ok, i see that. “not to well documented” doesn’t help me much -
i’d be coming back here to ask what is supposed to go in all those fields if I want to run a shell script. inside that I’ll send an email.

7

Please explain to us how it handles sessions, then. We need to be on the same page to be able to properly help you.

Yes, they were pointing to lb_policy.

8

A session is created in Tomcat… on 1 server.
If I switch to another server, I don’t have a valid session on THAT server… regardless of cookie or SESSIONID.
So you can’t help on this issue.

9

I looked at this section (and the single example).
No discussion that pertains to me.

→ So I’m unsure how to setup a system where its not so much PRIMARY and SECONDARY, but something more like CURRENT and BACKUP. And then when the health of the CURRENT is bad, the BACKUP starts getting traffic and IT becomes the CURRENT.
I’d like to correct the issue on the down server and get it back online, but don’t want traffic to go back to it UNLESS the health of the NOW CURRENT server goes bad, and on and on.
I’m wondering if the API is the only way to do this.

This is the concept I’m trying to explore and asking for discussion on.
If there’s no way to do this, even using the API, then say its impossible with Caddy.

10

Alright, so the session data is stored in memory, and the session is identified with a cookie. That makes sense.

My recommendation is that you implement session persistence in your Tomcat app to write the session data to a database. That’s the standard way of implementing high availability. See this article which I found which explains how to do this for Tomcat:

Doing it this way would make your service much more resilient to outages, in a way that any reverse proxy could not improve on their own. If you want high availability, you need to make session storage persistent across all instances of the application.

I don’t appreciate your tone. We’re providing free support. We’re volunteers. We need to fully understand your situation to be able to specifically recommend a solution. And that’s why we ask clarifying questions.

It’s not impossible to do with Caddy. But it’s true that none of the current selection policies do what you want on their own, right now. What you’d want is a sticky policy (i.e. cookie) paired with the first policy. Right now, cookie randomly picks when there’s no cookie, but a new policy could be created, let’s say cookie_first which is effectively the same thing as cookie but replaces the initial random selection with “first available” instead.

If you need this for your business, then I suggest you get a support plan to prioritize development of that feature: https://caddyserver.com/business

1 Like
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.