Double reverse proxy relay of primary domain to endpoint

1. Caddy version (caddy version):

2.45

a. System environment:

ubuntu focal

my issue

I have caddy running on two machines. One on aws lightsail instance

https://files.mydomain.org {
    import r53
    reverse_proxy /* https://238.files.mydomain.org {
     header_up Host {upstream_hostport}
     header_up X-Forwarded-Host {host}
      }
  }

and one at my router of my lan.

https://238.files.mydomain.org  {
  import r53
  reverse_proxy /* http://server.mydomain.net:11080
   }

I have an endpoint on a server on the LAN that needs to be reverse proxied as you can see above

My issue is the original domain is not being forwarded to the endpoint. The endpoint thinks https://238.files.mydomain.org is making the request instead of https://files.mydomain.org

That wouldn’t necessarily be an issue but the the app at that endpoint (nextcloud) won’t trust the RP because it’s not the domain expected https://files.mydomain.org but this “intermediate domain” https://238.files.mydomain.org. I can trust that one with nextcloud but then it rewrites the domain to this and I can’t have that.

So I’m thinking it’s this one I have to modify but I’m not sure how. Some header up down? a rewrite of the domain?

https://238.files.mydomain.org  {
  import r53
  reverse_proxy /* http://server.mydomain.net:11080
   }

You can probably do header_up Host {header.X-Forwarded-Host} to copy the host from the original request into the upstream request.

Also, remove the /* on each of your reverse_proxy lines. It’s not useful, it creates a tautological path matcher (the path will always start with a slash so there’s no use matching for that). It makes Caddy do more work than it has to (by like a few nanoseconds maybe).

couple questions

  1. So how does that differ from what I had
    header_up X-Forwarded-Host {host}
  2. Was this intented to be in both RPs or only in the one to the endpoint?

I’m talking about using that on your back Caddy server, not the front one which the browser hits first.

The first server sends X-Forwarded-Host to the back one, then the back one overrides Host with the X-Forwarded-Host received from the front one.

Thx! that was what it took to get those working but I have one more case which involves yet a third caddy server! inside the LAN serving a static Hugo Site and that’s not working with just passing the Host upstream like the others

So the three stanzas are below. Do I need maybe a rewrite at the file server? If so how to grab the passed down host?. Plus isn’t rewrite for the path not the host? When you build hugo you do it with a set url which has to match the url host/domain used or it fails to load assets scripts etc. I suppose I might be able to build with the last host/domain but I think it would be cleaner to make caddy translate/rewrite so hugo site can be built with the host the visitor uses.

  1. visitor facing
https://www.mydomain.org {
    import r53
    reverse_proxy https://238.www.mydomain.org {
      header_up Host {upstream_hostport}
      header_up X-Forwarded-Host {host}
      }
  }
  1. at LAN router
https://238.www.mydomain.org {
    import r53
    reverse_proxy https://deploy.mydomain.org {
      header_up Host {header.X-Forwarded-Host} 
     }
    }
  1. At static server (Hugo built static site built with base url https://www.mydomain.org )
https://deploy.mydomain.org {
     import r53
     root * /data/deploy
     file_server
     handle_errors {
      rewrite * /{http.error.status_code}.html
	  file_server
      }
     }

Since you’re doing HTTPS again from this one over, you shouldn’t override Host with the original domain, it has to be {upstream_host} otherwise TLS won’t work. But you don’t need to set X-Forwarded-Host again because it’ll be passed through (in v2.4.6 anyways, but in v2.5.0 you may need to make additional changes to make sure the upstream is trusted, see reverseproxy: Implement trusted proxies for `X-Forwarded-*` headers by francislavoie · Pull Request #4507 · caddyserver/caddy · GitHub)

so does that apply to 2.4.5 as well? I actually just tried 2.4.6 on my arm64 router and had a memory leak so had to revert.

Tada that worked! You @francislavoie and @matt are the BEST. I don’t need help very often but when I do you two always come through. I sure hope you have enough sponsors now to make a living as this is clearly one of the best most useful open source projects ever. I had an unusual IP routing issue and I/We solved it with Caddy!

https://238.www.mydomain.org {
    import r53-gpcca
    reverse_proxy https://deploy.mydomain.org {
     header_up Host {upstream_host}
      }
    }
1 Like

Yes.

But I’d be really surprised there’s a memory leak in 2.4.6 but not in 2.4.5. We’ve had no other similar reports since 2.4.6 was released.

Please make sure to fill out the details requested on the github issue.

:blush:

There’s definitely not enough sponsors yet.

1 Like

I think people should sponsor Francis too, he does a lot of the heavy lifting with the community and patches too!

I don’t want sponsorships! And I haven’t set it up on my account anyways. The money should go to @matt. I don’t need it.

As I mentioned endpoints are working. But I’m getting some response errors. r.

So do I need any header_down entries? My router firewall lets any port outbound through. Do those responses have to travel back to that first caddy instance at AWS lightsail? or do they get routed to the IP shown directily. If former then maybe I have to open up a bunch of ports at lightsail instance which would be maybe bad. Just a bit clueless how responses are handled that might involve web socket ports (which is what I assume those are?)

at my router

Feb 02 08:37:50 router caddy[12230]: {"level":"error","ts":1643819870.2433317,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 10.0.0.1:443->106.11.158.35:27240: write: broken pipe"}
Feb 02 08:47:39 router caddy[12230]: {"level":"error","ts":1643820459.2609708,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 10.0.0.1:443->114.119.145.66:59730: write: broken pipe"}
Feb 02 08:55:26 router caddy[12230]: {"level":"error","ts":1643820926.2230718,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 08:55:29 router caddy[12230]: {"level":"error","ts":1643820929.5755541,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 09:00:07 router caddy[12230]: {"level":"error","ts":1643821207.2664323,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 10.0.0.1:443->185.191.171.20:8192: write: broken pipe"}
Feb 02 09:02:39 router caddy[12230]: {"level":"error","ts":1643821359.936474,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"write tcp 10.0.0.1:443->106.11.156.76:17515: write: broken pipe"}
Feb 02 09:22:46 router caddy[12230]: {"level":"error","ts":1643822566.8292458,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 09:44:41 router caddy[12230]: {"level":"error","ts":1643823881.5220153,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 09:45:06 router caddy[12230]: {"level":"error","ts":1643823906.6379797,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}

at the first instance at AWS lightsail

Feb 02 16:27:17 awsgpcca caddy[27467]: {"level":"error","ts":1643819237.6308122,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 16:31:03 awsgpcca caddy[27467]: {"level":"error","ts":1643819463.8976974,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 16:31:51 awsgpcca caddy[27467]: {"level":"error","ts":1643819511.1487591,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 16:32:05 awsgpcca caddy[27467]: {"level":"error","ts":1643819525.2367232,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 16:32:12 awsgpcca caddy[27467]: {"level":"error","ts":1643819532.967717,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 16:55:26 awsgpcca caddy[27467]: {"level":"error","ts":1643820926.2085109,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 16:55:29 awsgpcca caddy[27467]: {"level":"error","ts":1643820929.5608206,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 17:22:46 awsgpcca caddy[27467]: {"level":"error","ts":1643822566.8140965,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 17:44:41 awsgpcca caddy[27467]: {"level":"error","ts":1643823881.5066693,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}
Feb 02 17:45:06 awsgpcca caddy[27467]: {"level":"error","ts":1643823906.6216168,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","error":"context canceled"}

I think those errors just mean that the client closed the connection.

Another small issue. It seems with this “relay” the original posters IP is lost and all I see in my webapp is the IP running that primary caddy instance.
So assume there is the poster’s IP in the header? Can that be forwarded with each upstream request?

SO. I’ve looked around a bit for a really complete tutorial/explanation of the standardized request header and all keys that can possibly be in one but not really finding anything (suggestion). So can caddy modify any key/value in the header or is it limited to some subset. If I can become a request header guru then I won’t have to keep asking questions like these.

It should be included in the X-Forwarded-For header. Your app might not be parsing that header correctly, it will have multiple values in that header, separated by commas.

This topic was automatically closed after 30 days. New replies are no longer allowed.