Reverse proxy behaving as a redirect

Help
1

1. The problem I’m having:

I migrated a server from 10.1.0.172:81 to 10.1.1.1:80 and as part of having a smooth transition I setup a Caddy server to catch requests to the old IP address.

The new address is in a different network range, so I also created a reverse proxy in case people have not updated the routing on their VPNs to access 10.1.1.x

Its worth mentioning that the new server address 10.1.1.1:80 is also served by Caddy.

2. Error messages and/or full log output:

Accessing 10.1.0.172:81 always results in a redirect to 10.1.1.1.

Accessing 10.1.0.172:80 within the network results in a proxied webpage appearing as http://10.1.0.172

$ curl -IL 10.1.0.172:80
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Tue, 03 Sep 2024 07:25:36 GMT
Location: http://10.1.1.1/login?back_url=http%3A%2F%2F10.1.1.1%2F
Referrer-Policy: strict-origin-when-cross-origin
Server: Caddy
Server: Caddy
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 714a4094-6d77-4e09-b37e-88f20bfe7fc5
X-Runtime: 0.006664
X-Xss-Protection: 1; mode=block

HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Length: 0
Content-Type: text/html; charset=utf-8
Etag: W/"5fbb2a59527060cc89b27ab4bb3e8a1d"
Referrer-Policy: strict-origin-when-cross-origin
Server: Caddy
Set-Cookie: _redmine_session=Nll0VU05TURma3IvU2hVVlRMTmZBK3ZWVEZ4TGFYWlBKbFdvUGkrTUtvUTQzbmxpakExM3pCSW5qVno2RXNYdlVUSEFVbEtJOUtTQ2lWb2Rhb2xldGxWREhwdVUrN3VBRGxJRnlxMmRqM2laeEV1YkxPVzc4d2NEQ09QTS9LR0dhVGhpRG5Db2x1bUdtKzNaalJCTUdnaysydjcycVNOSFVhQ0hDQXphWnpCOXV3Rk5aY1pQYjhKMVB4cy9naVRJLS1kWUNKRXJvYXNvWGZzWFVHWUJOT2RBPT0%3D--04c5525095c2d3ce2403c4020aba670f85c6e579; path=/; HttpOnly; SameSite=Lax
Vary: Accept
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: d5849379-4145-4476-b841-5ed18a42a451
X-Runtime: 0.197651
X-Xss-Protection: 1; mode=block
Date: Tue, 03 Sep 2024 07:25:36 GMT

Accessing 10.1.0.172:80 from the VPN results in a redirect to http://10.1.1.1

$ curl -IL 10.1.0.172:80
HTTP/1.1 302 Found
Cache-Control: no-cache
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Tue, 03 Sep 2024 07:22:53 GMT
Location: http://10.1.1.1/login?back_url=http%3A%2F%2F10.1.1.1%2F
Referrer-Policy: strict-origin-when-cross-origin
Server: Caddy
Server: Caddy
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: a33b361f-f7b8-4d8c-a682-5be3d97afb8a
X-Runtime: 0.025048
X-Xss-Protection: 1; mode=block

HTTP/1.1 200 OK
Cache-Control: max-age=0, private, must-revalidate
Content-Length: 0
Content-Type: text/html; charset=utf-8
Etag: W/"fc7a15cbde47db36216bbf9b4339eb77"
Referrer-Policy: strict-origin-when-cross-origin
Server: Caddy
Set-Cookie: _redmine_session=Yy92WkUxekpVYVBJb0pZTWJjUWJMMFM0SVFUc2tNci9ZZEVrck8zR0habWNZTXo4YlZxdHVvbUlIcWdpYlpjcHRLTmZxWUZBcTQ1cEhyTWg1bExXUkk2aDE3UHVmQXB2aTRDR2tUd3htT25ubHdwTWRBSExCN1p0dHcvUEpGOThFWC9HQzJzVkxyVXAyUDk4d0JoWTBuVVQ2cjVURlBRNUFwSUs2M2FWaG9xMklyUkpPeGpHbmRqb3VHK1JEYnp1LS14OS9qait5bDJpK1RNWlhXR2V4YkZBPT0%3D--a0a7a9cf52557153af67cac656baff9454a1e3f6; path=/; HttpOnly; SameSite=Lax
Vary: Accept
X-Content-Type-Options: nosniff
X-Download-Options: noopen
X-Frame-Options: SAMEORIGIN
X-Permitted-Cross-Domain-Policies: none
X-Request-Id: 59bb10d8-c2d4-48b9-9f84-0e0cefd81d27
X-Runtime: 0.302681
X-Xss-Protection: 1; mode=block
Date: Tue, 03 Sep 2024 07:22:54 GMT

Why is it redirecting and not proxying over VPN?

3. Caddy version:

Caddy v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

Docker

a. System environment:

Ubuntu 22.04.4 LTS
Docker version 27.1.2, build d01f264

b. Command:

c. Service/unit/compose file:

services:
  caddy:
    image: caddy:alpine
    restart: unless-stopped
    ports:
      - 10.1.0.172:80:80
      - 10.1.0.172:81:81
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile

d. My complete Caddy config:

{
        auto_https off
}

http://10.1.0.172:81 {
        redir http://10.1.1.1:80
}

http://10.1.0.172:80 {

        handle /* {
                reverse_proxy http://10.1.1.1:80 {
                        header_up Host {header.X-Forwarded-Host}
                }
        }

        handle_errors {
                respond "{err.status_code} {err.status_text}"
        }
}
2

I added debug to globals. When I browse over VPN to 10.1.0.172:80 I’m redirect to 10.1.1.1 and Caddy logs

2024/09/03 08:37:05.881 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "10.1.1.1:80", "total_upstreams": 1}
2024/09/03 08:37:05.894 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "10.1.1.1:80", "duration": 0.013160169, "request": {"remote_ip": "10.1.0.138", "remote_port": "39944", "client_ip": "10.1.0.138", "proto": "HTTP/1.1", "method": "GET", "host": "", "uri": "/", "headers": {"Accept": ["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/png,image/svg+xml,*/*;q=0.8"], "Cookie": ["REDACTED"], "X-Forwarded-For": ["10.1.0.138"], "X-Forwarded-Proto": ["http"], "Priority": ["u=0, i"], "Accept-Encoding": ["gzip, deflate"], "Upgrade-Insecure-Requests": ["1"], "X-Forwarded-Host": ["10.1.0.172"], "User-Agent": ["Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:129.0) Gecko/20100101 Firefox/129.0"], "Accept-Language": ["en-US,en;q=0.5"]}}, "headers": {"Cache-Control": ["no-cache"], "Location": ["http://10.1.1.1/login?back_url=http%3A%2F%2F10.1.1.1%2F"], "X-Download-Options": ["noopen"], "X-Permitted-Cross-Domain-Policies": ["none"], "X-Xss-Protection": ["1; mode=block"], "Content-Length": ["121"], "Content-Type": ["text/html; charset=utf-8"], "Referrer-Policy": ["strict-origin-when-cross-origin"], "X-Content-Type-Options": ["nosniff"], "Date": ["Tue, 03 Sep 2024 08:37:05 GMT"], "Server": ["Caddy"], "X-Frame-Options": ["SAMEORIGIN"], "X-Request-Id": ["65b7784b-6181-4246-96dd-6c2da69c6bb9"], "X-Runtime": ["0.007562"]}, "status": 302}
1 Like
3

Howdy @delanym, welcome to the Caddy community!

It seems to me like it IS proxying over the VPN. It dialled the upstream server 10.1.1.1:80 and forwarded the HTTP request, and that server responded with a redirect to a login page:

It’s not the front Caddy that’s generating this redirect, it’s the upstream one at 10.1.1.1.

You can see this in the debug roundtrip: a successful request and response back, with the response containing the Location redirect.

1 Like
4

The upstream Caddyfile

10.1.1.1:80 {
        request_body 1GB
        encode zstd gzip

        handle /* {
                reverse_proxy redmine:3000
        }

        handle_errors {
                respond "{err.status_code} {err.status_text}"
        }
}
5

Seems to me like you could strip the outer handle and just use reverse_proxy redmine:3000 - the matcher /* hits on all requests, there’s only one directive inside it, and you’re not making use of mutual exclusivity, so the presence of handle doesn’t actually affect any routing at all right now.

Other than that, I’m not seeing anything out of the ordinary, just another Caddy reverse-proxy. Nothing that would generate a redirect to a login page. That’s usually the provision of the upstream application.

1 Like
6

Ok i see what’s happening. The address redmine:3000 immediately redirects to a login page, returned in the location header. That address “10.1.1.1” is set in the redmine application itself. I could change it to “redmine:3000” (will try later). Then the first proxy should rewrite that location to 10.1.1.1. But why would it, since the second proxy is not rewriting 10.1.1.1 to 10.1.0.172?

I see proxying redirect responses is “a thing” Reverse Proxy and HTTP Redirects — Apache Traffic Server 9.2.5 documentation

The documentation for this module makes no specific mention of how redirects are handled reverse_proxy (Caddyfile directive) — Caddy Documentation

7

Change it instead to what the end user is supposed to browse to.

Caddy proxies try to be transparent, so they pass the upstream redirect all the way down the chain. You need the redirect to be correct in the context of the final client, e.g. the user’s browser.

In an ideal world, it would be better if your upstream was domain name agnostic and issued URI redirects instead of full URL redirects. Second to that, it would be better if your upstream composed URL redirects based on the Host header it receives, in which case you could configure Caddy to make sure those redirects are getting issued correctly. I’ve seen both of these behaviours fairly commonly. But when the upstream does neither, it’ll need to be manually configured.

Just to clarify on this front, there’s no documentation on how reverse_proxy handles redirects because it handles them like every other request, without any special exception. Client makes a request to Caddy, Caddy proxies upstream, upstream returns a response, Caddy passes the response to the client. There just happens to be two Caddy servers instead of one in the middle in your case.

It’s not particularly a thing - it’s just a technique used to wrangle backends that don’t play nice with Host and insist on URL redirects instead of URI redirects.

The “map rule” in this document is simply a header manipulation going upstream to the backend. The “reverse-map” rule is a header manipulation going downstream to the client.

You can make these manipulations with header_up and header_down inside your reverse_proxy.

1 Like
8

What the end user is supposed to browse to is different in the case where the originating IP is the second proxy handling requests at 10.1.0.172:80, since users hitting this address have no access to the 10.1.1.x/23 network range.

Well I don’t understand how its not a thing since Caddy must be able to distinguish an internal from an external redirect, since the former will need to be rewritten and the latter left as is.

I have added this config to the 2nd proxy to specifically handle redirects.

header_down Location "10.1.1.1/(.*)" "10.1.0.172/$1"

It is working! Thank you. But it still feels like something is not quite right.
If I proxy an address that address should be translated automatically.

9

It is a thing you can do, it’s just not “a thing” insofar as Apache Traffic Server has developed and named a whole feature set around it, but it’s not actually a unique feature. It’s just changing the headers going up and coming back down, that’s all. Caddy can do this, it just doesn’t need to call it some special feature with a unique name; you just modify the header however you need with the existing subdirectives that do that.

I don’t think that’s a good default, because Caddy doesn’t know when this might be a perfectly legitimate external URL redirect.

Caddy’s reverse proxy passes the Host header through unmodified, so the upstream knows which hostname it’s being accessed on if it only checked the Host header. If it knows it’s being accessed via one hostname, and it redirects you to a different hostname - how is Caddy supposed to tell that this isn’t a deliberate act? Websites redirect to login pages on different hosts all the time. I do it for all of my websites, because I use Authelia forward authentication and that bounces unauthenticated traffic to itself to login and then redirects back to the original resource. Changing the default to automatically interpret this as requiring a translation would break that kind of behaviour, for example.

Generally when faced with this kind of issue of selecting defaults, we want to opt for one that allows users to positively configure additional functionality they need, rather than requiring the other set of users to add configuration to disable the functionality they don’t need. Not to mention we’d be responsible for figuring out the specific logic that makes the most sense as a default for Caddy to set up these translations - which is another hurdle that would require careful thought and consideration and probably won’t cover all use cases. It’s much cleaner to let users configure Caddy for the use case they are fully aware of rather than have the developers guess a best-fit answer.

1 Like
10

The issue was that the host self-address was configured as 10.1.1.1. It should have been redmine:3000. When this is fixed the 2nd proxy no longer needs to rewrite the Location header on the way down because that is done by the 1st proxy. The 1st proxy was not doing this because it recognized 10.1.1.1 in the response from the server as an external address and so didn’t rewrite it. I don’t know enough to say how it knows this, but the fact is that it does. A reverse proxy that could not distinguish internal from external redirects would be pretty useless otherwise.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.