Infinite loop when HTTP_PROXY set

[louietan (Lei Tan)]

1. The problem I’m having:

I use Caddy as my local dev server, my setup breaks after upgrading Caddy from 2.6.4 to 2.7.4.
The request sent to the address proxied by Caddy results in Caddy infinitely loops back to itself.

2. Error messages and/or full log output:

*   Trying 127.0.0.1:80...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> GET / HTTP/1.1
> Host: localhost
> User-Agent: curl/7.68.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 503 Service Unavailable
< Content-Type: text/plain
< Date: Sat, 19 Aug 2023 12:14:03 GMT
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Server: Caddy
< Transfer-Encoding: chunked
<
Maximum number of open connections reached.
* Connection #0 to host localhost left intact

There were way more repeated < Server: Caddy lines than I pasted here, I removed them for brevity.
Caddy returned 503 because the HTTP proxy server finally rejected further connections from Caddy as it reached its maximum connections.

3. Caddy version:

2.7.4

4. How I installed and ran Caddy:

1. I’m using the binary from https://github.com/caddyserver/caddy/releases/download/v2.7.4/caddy_2.7.4_windows_amd64.zip
2. I’ve started an HTTP proxy server (Privoxy) on port 8118
3. And I’ve set environment variable HTTP_PROXY to http://127.0.0.1:8118

a. System environment:

OS: Windows 10 Pro x64

b. Command:

caddy run --watch Caddyfile

d. My complete Caddy config:

http://localhost {
        reverse_proxy :8080
}

5. Links to relevant resources:

The following 2 workarounds both works for me:

1. Specify the upstream host: reverse_proxy 127.0.0.1:8080
2. Override the Host header:

   http://localhost {
   	reverse_proxy :8080 {
   		header_up Host 127.0.0.1:8080
   	}
   }
   

For an HTTPS setup, Caddy returns status 308 with location of the requesting address, resulting infinite redirection.

[francislavoie]

@louietan

reverse_proxy :8080

@Mohammed90 pointed out to me that this is the problem; omitting the host in the upstream address is effectively the same as 0.0.0.0 which doesn’t make sense as a client address (as a server listen address it does mean “all IPv4 interfaces” though); it just happens to work as “localhost”.

I don’t know what your proxy is doing and if it misbehaves because of that. But I’d call that invalid config anyway so any of your suggested workaround are better than that.

[Mohammed90 (Mohammed Al Sahaf)]

Alright, it took me a while, but I have an explanation!

First, you have to know that Caddy passes the original Host header to upstream instead of using the upstream’s own address in the upstream request. This is critical to understand what’s happening here.

Second, the behavior of the Go runtime when it comes to handling the HTTP_PROXY address includes this caveat:

As a special case, if req.URL.Host is “localhost” (with or without a port number), then a nil URL and nil error will be returned.

Meaning, it checks if the dialed URL (the upstream, not the proxy) is localhost, 127.x.x.x, or the IPv6 representation of loopback; and if the upstream URL is any of those, the Go runtime will not pass it through the URL specified in HTTP_PROXY. This is why these workarounds work

@louietan

1. Specify the upstream host: reverse_proxy 127.0.0.1:8080
2. Override the Host header:

http://localhost {
	reverse_proxy :8080 {
		header_up Host 127.0.0.1:8080
	}
}

But why is the issue happening in the first place?
Well, with this config…

@louietan

http://localhost {
        reverse_proxy :8080
}

… this is the request journey:
1- Caddy receives a request on localhost:80
2- The dialed upstream address is 0.0.0.0:8080, so the Go runtime uses the URL specified in the HTTP_PROXY.
3- Caddy passes the original Host header
4- The proxy server receives the request with the value of the Host being localhost:80, per Caddy’s default behavior
5- The proxy server calls the service specified in the Host header, which is localhost:80, which is Caddy
6- Go back to 1.

My point to Francis still stands. The address 0.0.0.0 works by accident. It’s best to be explicit, especially now that we’ve seen the complexity it entails, but that could be me

[louietan (Lei Tan)]

Mohammed90 thanks for the detailed explanation.

Yeah, that :8080 is not a valid upstream makes sense to me.

After my confusion has been cleared, the more sensible workaround seems to me is to override the Host header with the upstream host, otherwise it still loops infinitely when proxying a remote host:

api.localhost {
	reverse_proxy http://api.remotehost.tld {
		# Without this, the Host header in the request is still "api.localhost",
		# when the request goes through the proxy server, the proxy forwards the
		# request to "api.localhost", which is Caddy, and now begins the loop...
		header_up Host {upstream_hostport}
	}
}

This has been mentioned here about SNI in TLS, the same principle applies when there is HTTP_PROXY.