Reverse-proxy request not seen on second server

1. The problem I’m having:

I am trying to setup a caddy reverse proxy that routes requests to other machines. I have 2 EC2 (Machine A, Machine B) instances with public IPs. I launched Caddy and have the following reverse proxy configs.

Machine A (52.87.249.192)

{
	debug
	on_demand_tls {
		ask http://127.0.0.1:9123
		# every 2 min we can sign 5 certificates
		burst 5
		interval 2m
	}
}

:443 {
	reverse_proxy localhost:8000
	tls {
		on_demand
	}
}

:80 {
  reverse_proxy http://3.235.124.246
}

Machine B (3.235.124.246)

{
    debug
    on_demand_tls {
      ask http://127.0.0.1:9123
      # every 2 min we can sign 5 certificates
      burst 5
      interval 2m
    }
}

http://3.235.124.246:80 {
	reverse_proxy localhost:9000
  respond /caddy-health-check 200 {
    body "Caddy http reverse proxy running!
"
  }
}

If I run

curl -v http://52.87.249.192/<anything I want here> 

on my local machine, I always get a 200 response with no content. But on the Caddy logs on Machine B or the server listening to port 9000 on Machine B, I don’t see anything.

If I run

curl -v http://3.235.124.246/caddy-health-check

on my local machine or on Machine A, I get back the expected “Caddy http reverse proxy running!” response.

If I run

curl -v http://3.235.124.246/<anything I want here> 

on either my local machine or Machine A, I see the request on both Caddy logs and the server listening to 9000 on Machine B.

What I would like to do is to be able to send any request from my local machine to Machine A and forward the exact request to Machine B and then have the response sent back to my local machine.

Any help would be appreciated. Thanks!

2. Error messages and/or full log output:

I don’t actually have error messages.
On Machine A, whenever I make a request the Caddy logs prints this:

2024/07/02 17:50:00.231 DEBUG   http.handlers.reverse_proxy     selected upstream       {"dial": "3.235.124.246:80", "total_upstreams": 1}
2024/07/02 17:50:00.236 DEBUG   http.handlers.reverse_proxy     upstream roundtrip      {"upstream": "3.235.124.246:80", "duration": 0.004377609, "request": {"remote_ip": "143.58.200.28", "remote_port": "55204", "client_ip": "143.58.200.28", "proto": "HTTP/1.1", "method": "GET", "host": "52.87.249.192", "uri": "/caddyajlfk", "headers": {"User-Agent": ["curl/8.4.0"], "Accept": ["*/*"], "X-Forwarded-For": ["143.58.200.28"], "X-Forwarded-Proto": ["http"], "X-Forwarded-Host": ["52.87.249.192"]}}, "headers": {"Server": ["Caddy"], "Date": ["Tue, 02 Jul 2024 17:50:00 GMT"], "Content-Length": ["0"]}, "status": 200}

But anything going through the reverse proxy on Machine A does not show up on logs on Machine B.

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

a. System environment:

I installed go and xcaddy through a nix flake and then built caddy with

xcaddy build --with github.com/caddy-dns/godaddy

Although technically I no longer need the godaddy version.

b. Command:

I start caddy by running

caddy start --config Caddyfile

c. Service/unit/compose file:

d. My complete Caddy config:

I pasted both my caddy configs in the question above.

5. Links to relevant resources:

Howdy @Aditya_Rajagopal, welcome to the Caddy community! Thanks for filling out the Help template and in particular providing those debug logs - they will help show what’s going on.

So, in order to explain why this is happening and how to fix it, first I need to demonstrate a specific nuance when it comes to Caddy server: Caddy responds to UNCONFIGURED requests with a Status 200 (OK). For a little bit of background on this, you can have a read of this informational post: Why Caddy emits empty 200 OK responses by default

To quickly show you what this means, though, I can start Caddy with this Caddyfile:

http://localhost {
  respond 403
}

And try to curl this address:

➜ curl -I localhost
HTTP/1.1 403 Forbidden
Server: Caddy
Date: Wed, 03 Jul 2024 05:16:11 GMT

A response of 403, exactly what we expected. But if we try to access it via the loopback IP address…

➜ curl -I 127.0.0.1
HTTP/1.1 200 OK
Server: Caddy
Date: Wed, 03 Jul 2024 05:16:07 GMT

A response of 200. Why?

Because in the Caddyfile, I have only configured respond 403 for http://localhost. I did not tell Caddy what I want it to do for requests to http:///127.0.0.1. Therefore, since it has no configuration for that site, it will respond with an empty status 200.

If I wanted BOTH to give 403 responses, I would need to add the second site address - so, I’d have http://localhost, http://127.0.0.1 in my Caddyfile. Or, if I wanted it to respond 403 for ANY hostname over HTTP, I could use a site address of just http://.

Now, with that information in mind, lets have a look at your http.handlers.reverse_proxy debug log:

{
  "upstream": "3.235.124.246:80",
  "duration": 0.004377609,
  "request": {
    "remote_ip": "143.58.200.28",
    "remote_port": "55204",
    "client_ip": "143.58.200.28",
    "proto": "HTTP/1.1",
    "method": "GET",
    "host": "52.87.249.192",
    "uri": "/caddyajlfk",
    "headers": {
      "User-Agent": [
        "curl/8.4.0"
      ],
      "Accept": [
        "*/*"
      ],
      "X-Forwarded-For": [
        "143.58.200.28"
      ],
      "X-Forwarded-Proto": [
        "http"
      ],
      "X-Forwarded-Host": [
        "52.87.249.192"
      ]
    }
  },
  "headers": {
    "Server": [
      "Caddy"
    ],
    "Date": [
      "Tue, 02 Jul 2024 17:50:00 GMT"
    ],
    "Content-Length": [
      "0"
    ]
  },
  "status": 200
}

Caddy responded with an empty 200 status. This is the same response we expect when Caddy is UNCONFIGURED for this route. Why?

Take a look at the request.host in the debug log: “52.87.249.192”. When you made a curl request to the first Caddy via its IP address, the first Caddy contacted the upstream Caddy and passed along the hostname you originally requested - that is, the IP address.

But you haven’t configured the upstream Caddy to respond to requests for this IP address. You have only configured the upstream Caddy to respond to requests for the host 3.235.124.246 over HTTP on port 80.

So how do you fix it?

There’s two ways to fix this issue. You can either configure the first Caddy to tell the upstream Caddy what it wants to see: that involves overriding the Host header being sent upstream, so that the upstream Caddy sees you requesting 3.235.124.246.

Or, you can configure the upstream Caddy to accept and respond to requests for 52.87.249.192. This might seem a little unintuitive since the upstream Caddy isn’t listening on this IP, but all it’s really doing is matching the requested Host.

Alternatively - you can sidestep this little nuance of IP addresses by using actual hostnames; domain names that actually resolve to IPs. Regardless of the IP address that’s being connected to, the original domain name that your client requests is going to become the Host header that carries through to the end. So if both Caddy servers were configured with the site mydomain.example.com and you reverse-proxy from one to the other, it’ll just work, regardless of IPs.

1 Like

Hey @Whitestrake, thanks for getting back! That fixed the issue! Adding Machine A’s IP to the pattern to match on on Machine B fixed the issue.

2 Likes

Sorry I’m late here, had a busy week. In addition to what Matthew said:

Remove this, these options are deprecated and will soon be removed. They don’t work properly at all anymore.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.