Reverse Proxy - continual 308 replies

1. Caddy version (caddy version):

v2.1.1 h1:X9k1+ehZPYYrSqBvf/ocUgdLSRIuiNiMo7CvyGUQKeA=

2. How I run Caddy:

./caddy_linux_amd64 start

a. System environment:

CentOS 7 - VM with 5 network cards each with 1 ip address assigned (1.1.1.1, 2.2.2.2, 3.3.3.3, 4.4.4.4, and 5.5.5.5). The 6.6.6.6 ip address is an external ip address.

b. Command:

./caddy_linux_amd64 start

c. Service/unit/compose file:

n/a

d. My complete Caddyfile or JSON config:

1.1.1.1:80 {
        respond /is_ok "Ok"
        respond / "1.1.1.1"
        respond /loadbalancer "1.1.1.1"
        log {
                output file /var/log/caddy/caddy-1.1.1.1.log
        }
}

2.2.2.2:80 {
        respond /is_ok "Ok"
        respond / "2.2.2.2"
        respond /loadbalancer "2.2.2.2"
        log {
                output file /var/log/caddy/caddy-2.2.2.2.log
        }
}

3.3.3.3:80 {
        respond /is_ok "Ok"
        respond / "3.3.3.3"
        respond /loadbalancer "3.3.3.3"
        log {
                output file /var/log/caddy/caddy-3.3.3.3.log
        }
}

4.4.4.4:80 {
        respond /is_ok "Ok"
        respond / "4.4.4.4"
        respond /loadbalancer "4.4.4.4"
        log {
                output file /var/log/caddy/caddy-4.4.4.4.log
        }
}

www.example.com 5.5.5.5 {
        bind 5.5.5.5
        respond / "5.5.5.5"
        reverse_proxy /loadbalancer 1.1.1.1:80 2.2.2.2:80 3.3.3.3:80 4.4.4.4:80 {
        lb_policy round_robin
        lb_try_duration 10s
        lb_try_interval 1s
        health_path /is_ok
        health_status 200
        health_body "Ok"
        health_timeout 5s
        health_interval 30s
        }
        log {
                output file /var/log/caddy/caddy-master.log
        }
}

3. The problem I’m having:

After startup and waiting 30 seconds for the reverse proxies to become ‘up’, when attempting to visit https://www.example.com/loadbalancer I continually receive 308 requests when I expect to receive the IP address. Visiting http://1.1.1.1/loadbalancer produces the correct response - and the same for the other three ip addresses (2.2.2.2, 3.3.3.3, and 4.4.4.4). Visiting http://1.1.1.1/is_ok also produces the correct response - and the same for the other three ip addresses (2.2.2.2, 3.3.3.3, and 4.4.4.4).

4. Error messages and/or full log output:

2020/09/04 21:49:56.294 INFO    http.log.access handled request {"request": {"method": "GET", "uri": "/loadbalancer", "proto": "HTTP/1.1", "remote_addr": "2.2.2.2:60482", "host": "www.example.com", "headers": {"X-Forwarded-For": ["6.6.6.6"], "X-Forwarded-Proto": ["https"], "Accept-Encoding": ["gzip"], "User-Agent": ["insomnia/2020.3.3"], "Accept": ["*/*"]}}, "common_log": "2.2.2.2 - - [04/Sep/2020:17:49:56 -0400] \"GET /loadbalancer HTTP/1.1\" 308 0", "duration": 0.000054135, "size": 0, "status": 308, "resp_headers": {"Content-Type": [], "Server": ["Caddy"], "Location": ["https://www.example.com/loadbalancer"], "Connection": ["close"]}}
2020/09/04 21:49:56.323 INFO    http.log.access handled request {"request": {"method": "GET", "uri": "/loadbalancer", "proto": "HTTP/1.1", "remote_addr": "3.3.3.3:52562", "host": "www.example.com", "headers": {"User-Agent": ["insomnia/2020.3.3"], "Accept": ["*/*"], "X-Forwarded-For": ["6.6.6.6"], "X-Forwarded-Proto": ["https"], "Accept-Encoding": ["gzip"]}}, "common_log": "3.3.3.3 - - [04/Sep/2020:17:49:56 -0400] \"GET /loadbalancer HTTP/1.1\" 308 0", "duration": 0.000114372, "size": 0, "status": 308, "resp_headers": {"Location": ["https://www.example.com/loadbalancer"], "Content-Type": [], "Server": ["Caddy"], "Connection": ["close"]}}
2020/09/04 21:49:56.360 INFO    http.log.access handled request {"request": {"method": "GET", "uri": "/loadbalancer", "proto": "HTTP/1.1", "remote_addr": "4.4.4.4:45316", "host": "www.example.com", "headers": {"User-Agent": ["insomnia/2020.3.3"], "Accept": ["*/*"], "X-Forwarded-For": ["6.6.6.6"], "X-Forwarded-Proto": ["https"], "Accept-Encoding": ["gzip"]}}, "common_log": "4.4.4.4 - - [04/Sep/2020:17:49:56 -0400] \"GET /loadbalancer HTTP/1.1\" 308 0", "duration": 0.000053907, "size": 0, "status": 308, "resp_headers": {"Connection": ["close"], "Content-Type": [], "Server": ["Caddy"], "Location": ["https://www.example.com/loadbalancer"]}}

Log from Insomnia

* Preparing request to http://www.example.com/loadbalancer
* Current time is 2020-09-04T22:10:59.433Z
* Using libcurl/7.69.1-DEV OpenSSL/1.1.1d zlib/1.2.11 WinIDN libssh2/1.9.0_DEV nghttp2/1.40.0
* Using default HTTP version
* Disable timeout
* Enable automatic URL encoding
* Enable SSL validation
* Enable cookie sending with jar of 2 cookies
* Too old connection (1263 seconds), disconnect it
* Connection 33 seems to be dead!
* Closing connection 33
* TLSv1.3 (OUT), TLS alert, close notify (256):
*   Trying 5.5.5.5:80...
* Connected to www.example.com (5.5.5.5) port 80 (#34)

> GET /loadbalancer HTTP/1.1
> Host: www.example.com
> User-Agent: insomnia/2020.3.3
> Accept: */*

* Mark bundle as not supporting multiuse

< HTTP/1.1 308 Permanent Redirect
< Connection: close
< Location: https://www.example.com/loadbalancer
< Server: Caddy
< Date: Fri, 04 Sep 2020 22:10:59 GMT
< Content-Length: 0


* Closing connection 34
* Issue another request to this URL: 'https://www.example.com/loadbalancer'
*   Trying 5.5.5.5:443...
* Connected to www.example.com (5.5.5.5) port 443 (#35)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: C:\Users\Example\AppData\Local\Temp\insomnia_2020.3.3\2017-09-20.pem
*   CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=www.example.com
*  start date: Sep  4 14:13:50 2020 GMT
*  expire date: Dec  3 14:13:50 2020 GMT
*  subjectAltName: host "www.example.com" matched cert's "www.example.com"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x2dea3d138f0)

> GET /loadbalancer HTTP/2
> Host: www.example.com
> user-agent: insomnia/2020.3.3
> accept: */*

* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* Connection state changed (MAX_CONCURRENT_STREAMS == 250)!

< HTTP/2 308 
< date: Fri, 04 Sep 2020 22:10:59 GMT
< location: https://www.example.com/loadbalancer
< server: Caddy
< server: Caddy
< content-length: 0

* Received 0 B chunk
* Connection #35 to host www.example.com left intact
* Issue another request to this URL: 'https://www.example.com/loadbalancer'
* Found bundle for host www.example.com: 0x2dea6ac0960 [can multiplex]
* Re-using existing connection! (#35) with host www.example.com
* Connected to www.example.com (5.5.5.5) port 443 (#35)
* Using Stream ID: 3 (easy handle 0x2dea3d138f0)

> GET /loadbalancer HTTP/2
> Host: www.example.com
> user-agent: insomnia/2020.3.3
> accept: */*

< HTTP/2 308 
< date: Fri, 04 Sep 2020 22:10:59 GMT
< location: https://www.example.com/loadbalancer
< server: Caddy
< server: Caddy
< content-length: 0


* Received 0 B chunk
* Connection #35 to host www.example.com left intact
* Issue another request to this URL: 'https://www.example.com/loadbalancer'
* Found bundle for host www.example.com: 0x2dea6ac0960 [can multiplex]
* Re-using existing connection! (#35) with host www.example.com
* Connected to www.example.com (5.5.5.5) port 443 (#35)
* Using Stream ID: 5 (easy handle 0x2dea3d138f0)

> GET /loadbalancer HTTP/2
> Host: www.example.com
> user-agent: insomnia/2020.3.3
> accept: */*

< HTTP/2 308 
< date: Fri, 04 Sep 2020 22:10:59 GMT
< location: https://www.example.com/loadbalancer
< server: Caddy
< server: Caddy
< content-length: 0


* Received 0 B chunk
* Connection #35 to host www.example.com left intact
* Issue another request to this URL: 'https://www.example.com/loadbalancer'
* Found bundle for host www.example.com: 0x2dea6ac0960 [can multiplex]
* Re-using existing connection! (#35) with host www.example.com
* Connected to www.example.com (5.5.5.5) port 443 (#35)
* Using Stream ID: 7 (easy handle 0x2dea3d138f0)

> GET /loadbalancer HTTP/2
> Host: www.example.com
> user-agent: insomnia/2020.3.3
> accept: */*

< HTTP/2 308 
< date: Fri, 04 Sep 2020 22:10:59 GMT
< location: https://www.example.com/loadbalancer
< server: Caddy
< server: Caddy
< content-length: 0


* Received 0 B chunk
* Connection #35 to host www.example.com left intact
* Issue another request to this URL: 'https://www.example.com/loadbalancer'
* Found bundle for host www.example.com: 0x2dea6ac0960 [can multiplex]
* Re-using existing connection! (#35) with host www.example.com
* Connected to www.example.com (5.5.5.5) port 443 (#35)
* Using Stream ID: 9 (easy handle 0x2dea3d138f0)

> GET /loadbalancer HTTP/2
> Host: www.example.com
> user-agent: insomnia/2020.3.3
> accept: */*

< HTTP/2 308 
< date: Fri, 04 Sep 2020 22:10:59 GMT
< location: https://www.example.com/loadbalancer
< server: Caddy
< server: Caddy
< content-length: 0

5. What I already tried:

Spent better part of 2 days trying to work out a solution. Nothing has worked so far.

6. What I would expect:

When visiting http://5.5.5.5/loadbalancer I would receive a text of “1.1.1.1”, “2.2.2.2”, “3.3.3.3”, or “4.4.4.4” depending on the which backend proxy is chosen.

7. Links to relevant resources:

You’re getting hit by the HTTP->HTTPS redirect. Try making your request with https:// instead of http:// and it should work fine.

Edit: Scratch that. What’s happening is you’re proxying back to yourself, but Caddy passes through the Host header when you make a request with www.example.com, so Caddy routes the request back to your www.example.com site block instead of one of your load balancers.

This is partly because you have them all in the same Caddy instance. You should probably not do that lol

You can override this by setting the Host header to the upstream being used, as a subdirective to reverse_proxy. This should hopefully get around this redirect issue:

header_up Host {http.reverse_proxy.upstream.hostport}
1 Like

This topic was automatically closed after 30 days. New replies are no longer allowed.