How reverse proxy round-robin works?

[westwin]

1. My Caddy version devel with very latest code since Apr 17 2020

2. How I run Caddy:

I’m playing with caddy as a reverse proxy server.

As you can find below my caddyfile: caddy will reverse round-robin to two upstream services on port 9010/9020.

the reverse proxy works very well until I stopped the service on 9010 port. caddy can successfully reverse to 9020 port. The problem is that after I brought back 9010 to work, caddy can not route the traffic to 9010, all traffic goes to 9020. I didn’t find any document how to make it work ?

Thanks in advance

:2015 {
reverse_proxy localhost:9010 loccalhost:9020 {
lb_policy round_robin
lb_try_duration 100ms
lb_try_interval 250ms
}
}

a. System environment:

macos

[francislavoie]

I’ll need to dig into it a bit more to verify, but I think it’s because you’re not specifying any health checks for Caddy to detect that the backend is back up. Could you try enabling either active or passive health checks?

[matt (Matt Holt)]

Without health checks enabled, it shouldn’t ever mark a backend as down; it should just try it and move onto the next. Can anyone repro?

Edit: @westwin You’ve spelled it wrong:

loccalhost:9020

That will cause it to always fail.

[westwin]

that is just a typo

[westwin]

Is there any chance that caddy somehow caches the http connection, which always sticks on port 9020.

Because I added some debug log in httptransport.go, see below, the re-dial does not happen

rt := &http.Transport{
	DialContext: func(ctx context.Context, network, address string) (net.Conn, error) {
		// the proper dialing information should be embedded into the request's context
		fmt.Printf("dial again request: %s, %s\n", network, address)
		if dialInfo, ok := GetDialInfo(ctx); ok {
			network = dialInfo.Network
			address = dialInfo.Address
		}
		fmt.Printf("dial again: %s, %s\n", network, address)

And I confirmed that the traffic was not forwarded to port 9010 with a wireshark capture

[matt (Matt Holt)]

Hm, I suppose that might be possible – thanks for helping to troubleshoot this! I’ll be pretty busy this week, do you think you can drill down a little more to determine if that is the problem? Find a way to close the connection (it can be hacky just for your testing) and then see if it solves it?

[westwin]

Thanks Matt.

In case I removed the if req.URL.Host == “” check in method directRequest of reverseproxy.go, then it will work as a charm. (will re-dial again).

The if check does not meet since it’s already set to localhost:9010, but the dial happens on 9020(which will re-use the http-keeplived cache connection).

Is there any reason that why add the if check ?

[matt (Matt Holt)]

That… hmm… if I did know once, don’t know anymore

We can patch this pretty quick after the 2.0 release here in a week or two. Doing it after 2.0 will ensure we don’t break anything right before 2.0, and then people using the latest will have time to help test before the 2.1 release.

Anyway, nice find! Thanks for the help.

Want to submit a PR or should I do it?

[westwin]

If this is the root cause. I’d suggest you help submit a PR before I understood how to submit a PR

[westwin]

I just submitted a PR, please find reverseproxy: always set the req.URL.Host with the upstream by westwin · Pull Request #3290 · caddyserver/caddy · GitHub