Reverse proxy to S3 bucket with handle_response for 404s

steve1 · August 25, 2021, 5:12am

1. Caddy version (`caddy version`):

v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=

2. How I run Caddy:

caddy run

a. System environment:

Ubuntu 20.04.2 LTS on AWS EC2

b. Command:

caddy run

d. My complete JSON config:

{
	"apps": {
		"http": {
			"servers": {
				"blahblahblah": {
					"listen": [":443"],
					"routes": [
						{
							"match": [{
								"host": ["blahblahblah.com"]
							}],
							"handle": [{
								"handler": "vars",
								"site_id": "ABCDEFG123456"
							}]
						},
						{
							"handle": [
								{
									"handler": "vars",
									"bucket_address": "blahblahblah-content.s3-website-us-east-1.amazonaws.com"
								},
								{
									"handler": "encode",
									"encodings": {
										"gzip": {}
									},
									"minimum_length": 0
								},
								{
									"handler": "subroute",
									"routes": [
										{
											"group": "group0",
											"handle": [{
												"handler": "rewrite",
												"uri": "/{http.vars.site_id}{http.request.uri}"
											}]
										},
										{
											"handle": [{
												"handler": "reverse_proxy",
												"headers": {
													"request": {
														"set": {
															"Host": ["{http.vars.bucket_address}"]
														}
													}
												},
												"upstreams": [{
													"dial": "{http.vars.bucket_address}:80"
												}],
												"handle_response": [{
													"match": {
														"status_code": [404]
													},
													"routes": [{
														"handle": [
															{
																"handler": "rewrite",
																"uri": "/file/html/get?site_id={http.vars.site_id}&path={http.request.orig_uri}"
															},
															{
																"handler": "reverse_proxy",
																"headers": {
																	"request": {
																		"set": {
																			"Host": ["{http.request.orig_uri}"]
																		}
																	}
																},
																"upstreams": [{
																	"dial": "api.blahblahblah.com:80"
																}]
															}
														]
													}]
												}]
											}]
										}
									]
								}
							]
						}
					]
				}
			}
		}
	}
}

3. The problem I’m having:

I’m using Caddy as a reverse proxy to an S3 bucket. I use handle_response to call my API to generate a new file for the bucket if the file does not exist. It’s mostly working fine.

My issues are as follows:

I get the following errors whenever there’s a 404 for the bucket:

ERROR|http.handlers.reverse_proxy|reading from backend|{error: http: read on closed response body}|
ERROR|http.handlers.reverse_proxy|aborting with incomplete response|{error: http: read on closed response body}|

Is this something I need to worry about? Anything I can do to muffle those errors?

My upstream dial to api.blahblahblah.com only works with port 80 which I’m assuming is related to my initial dial to the S3 bucket by port 80. How do I dial using port 443?

francislavoie · August 25, 2021, 7:12am

This is a known issue with handle_response at this time.

github.com/caddyserver/caddy

handle_response errors with "read on closed response body"

opened 02:42AM - 22 Aug 21 UTC

closed 06:00PM - 18 Oct 21 UTC

polarathene

bug

Following the [docs for `reverse_proxy` directive with `handle_response`](https:…//caddyserver.com/docs/caddyfile/directives/reverse_proxy#intercepting-responses) on a very basic JSON response endpoint results in failure, but modifying the header elsewhere in Caddy works as expected. **Caddyfile (v2.4.3):** ```Caddyfile :80 { # Works: #header Cache-Control "max-age=10" reverse_proxy localhost:8000 { handle_response { # Errors: header Cache-Control "max-age=10" } # Works: #header_down Cache-Control "max-age=10" } } ``` **Error logged:** ``` 2021/08/22 02:34:00.645 ERROR http.handlers.reverse_proxy reading from backend {"error": "http: read on closed response body"} 2021/08/22 02:34:00.645 ERROR http.handlers.reverse_proxy aborting with incomplete response {"error": "http: read on closed response body"} ``` The upstream still received the request and responded no differently AFAIK, in the web browser network tools Caddy has sent a response header but omitted the body. --- For added context, intention was to match on `Content-Type application/json` rather than the URI; but AFAIK this is only available via the response, and thus can only be matched within `handle_response`? As the response matcher is not required to reproduce the error I have excluded it.

Just a warning, the thread is very long winded, but TL;DR:

The handle_response feature was not intended to be used in conjunction with responses that could have a request body, so if the response has a body at all these errors would happen.

It was more meant to be used with responses that would be empty and have special response headers that could be used to instruct the subsequent routes how to rehandle the request (like X-Accel-Redirect for example, as shown in the docs).

Some additional work will need to be done to make it work properly in situations where a response body is returned.

I think it’s because of the Host header you’re setting. That’ll mess up HTTPS proxying if it doesn’t match what the upstream espects. Also, you need to turn on the transport > http > tls option to tell Caddy to use TLS (setting tls to an empty object {} is sufficient to enable TLS).

steve1 · August 25, 2021, 4:07pm

Thanks Francis!

Should I not be using handle_response with a reverse_proxy in this manner? Is there a better way?

I think it’s because of the Host header you’re setting. That’ll mess up HTTPS proxying if it doesn’t match what the upstream espects. Also, you need to turn on the transport > http > tls option to tell Caddy to use TLS (setting tls to an empty object {} is sufficient to enable TLS).

"handle": [
	{
		"handler": "rewrite",
		"uri": "/file/html/get?site_id={http.vars.site_id}&path={http.request.orig_uri}"
	},
	{
		"handler": "reverse_proxy",
		"transport": {
			"protocol": "http",
			"tls": {}
		},
		"upstreams": [{
			"dial": "api.blahblahblah.com:443"
		}]
	}
]

Does this not look right? I get a 400 error: “The plain HTTP request was sent to HTTPS port”.

francislavoie · August 25, 2021, 4:27pm

Right now, unfortunately no. It’s not currently a supported usecase of handle_response, but it might be later once that github issue is resolved. I don’t have an alternative for you at this time.

You’re missing a line to set the Host header. It might look like this:

                {
                  "handler": "reverse_proxy",
                  "headers": {
                    "request": {
                      "set": {
                        "Host": [
                          "{http.reverse_proxy.upstream.hostport}"
                        ]
                      }
                    }
                  },
                  "transport": {
                    "protocol": "http",
                    "tls": {}
                  },
                  "upstreams": [
                    {
                      "dial": "example.com:443"
                    }
                  ]
                }

That placeholder will set the Host header to api.blahblahblah.com:443, i.e. the upstream hostport.

steve1 · August 25, 2021, 6:17pm

Thanks Francis- I tried that, but no change, I still get the “400 Bad Request - The plain HTTP request was sent to HTTPS port” error.

matt · August 25, 2021, 6:26pm

Make sure to reload your Caddy config after changing it.

francislavoie · August 25, 2021, 6:31pm

The only thing I can think of is that – you haven’t properly reloaded your config. The config I posted is definitely correct.

steve1 · August 25, 2021, 6:52pm

I reloaded, but no dice. :^)

{
	"apps": {
		"http": {
			"servers": {
				"testsite": {
					"listen": [":443"],
					"routes": [
						{
							"match": [{
								"host": ["example.com"]
							}],
							"handle": [{
								"handler": "vars",
								"site_id": "5ace6cb611704a7fbd849075"
							}]
						},
						{
							"handle": [
								{
									"handler": "vars",
									"bucket_address": "mybucket-content.s3-website-us-east-1.amazonaws.com"
								},
								{
									"handler": "encode",
									"encodings": {
										"gzip": {}
									},
									"minimum_length": 0
								},
								{
									"handler": "subroute",
									"routes": [
										{
											"group": "group0",
											"handle": [{
												"handler": "rewrite",
												"uri": "/{http.vars.site_id}{http.request.uri}"
											}]
										},
										{
											"handle": [{
												"handler": "reverse_proxy",
												"headers": {
													"request": {
														"set": {
															"Host": ["{http.vars.bucket_address}"]
														}
													}
												},
												"upstreams": [{
													"dial": "{http.vars.bucket_address}:80"
												}],
												"handle_response": [{
													"match": {
														"status_code": [404]
													},
													"routes": [{
														"handle": [
															{
																"handler": "rewrite",
																"uri": "/file/html/get?site_id={http.vars.site_id}&path={http.request.orig_uri}"
															},
															{
																"handler": "reverse_proxy",
																"headers": {
												                    "request": {
												                      "set": {
												                        "Host": [
												                          "{http.reverse_proxy.upstream.hostport}"
												                        ]
												                      }
												                    }
												                  },
																"transport": {
												                    "protocol": "http",
												                    "tls": {}
												                  },
																"upstreams": [{
																	"dial": "api.samplesite.com:443"
																}]
															}
														]
													}]
												}]
											}]
										}
									]
								}
							]
						}
					]
				}
			}
		}
	}
}

system · September 24, 2021, 5:13am

This topic was automatically closed after 30 days. New replies are no longer allowed.