Reverse_proxy got net/http: HTTP/1.x transport connection broken: http: ContentLength=79 with Body length 0

Help
1

1. Caddy version (caddy version): v2.2.0

2. How I run Caddy:

a. System environment:

b. Command:

paste command here

c. Service/unit/compose file:

paste full file contents here

d. My complete Caddyfile or JSON config:

[
    {
        "handler": "custom_hadnler"
    },
    {
        "handler": "reverse_proxy",
        "transport": {
            "dial_timeout": "10s",
            "expect_continue_timeout": "20s",
            "keep_alive": {
                "enabled": true,
                "idle_timeout": "60s",
                "max_idle_conns": 100,
                "max_idle_conns_per_host": 20
            },
            "max_response_header_size": 4096,
            "protocol": "http",
            "read_buffer_size": 4194304,
            "response_header_timeout": "20s",
            "write_buffer_size": 4194304
        },
        "upstreams": [{
            "dial": "xxx:80"
        }]
    }
]

3. The problem I’m having:

reverse_proxy return the following error log:

{
    "level": "debug",
    "ts": "2020-10-05 10:15:06.384",
    "logger": "http.handlers.reverse_proxy",
    "msg": "upstream roundtrip",
    "upstream": "xxx:80",
    "request": {
        "remote_addr": "[::1]:55080",
        "proto": "HTTP/1.1",
        "method": "POST",
        "host": "xxx",
        "uri": "/xxxxxx"
        "headers": {
            "User-Agent": [
                "curl/7.54.0"
            ],
            "Content-Type": [
                "application/x-www-form-urlencoded"
            ],
            "X-Forwarded-Proto": [
                "http"
            ],
            "X-Forwarded-For": [
                "::1"
            ],
            "Accept": [
                "*/*"
            ],
            "Postman-Token": [
                "04763d08-d8ff-43ed-8d84-30d8d37b1aa2"
            ],
            "Cache-Control": [
                "no-cache"
            ],
            "Content-Length": [
                "79"
            ]
        }
    },
    "duration": 0.052033278,
    "error": "net/http: HTTP/1.x transport connection broken: http: ContentLength=79 with Body length 0"
}

4. Error messages and/or full log output:

5. What I already tried:

The problem occurs for the following conditions:

  1. the request with header: application/x-www-form-urlencoded
  2. If the previous custom_handler did the ParseForm

then the reverse_proxy got this error, the reason should be the previous custom handler have read the body in the form format, so the reverse_proxy read the 0 length body.

In some scenarios, the previous handlers will do some work and need the form value, so is it possible for reverse_proxy to handle this special case?

2

Your custom handler will either need to also update the content length header on the request after consuming the form data, or it will need to wrap the request to make subsequent handlers be able to read from the request again.

Maybe something like this: http - How to read request body twice in Golang middleware? - Stack Overflow

1 Like
3

@francislavoie thank you for your suggestion.

Actually the previous custom handlers are basic the same as some web framework middleware, they may do some work (maybe including ParseForm), and then transfer the http.Request to the core business handler, since the ParseForm is idempotent, so nothing special to be handled. But for reverse_proxy, it changes, I wonder if this case can be handled in a unified way, thus there is no need for the the previous handlers or middlewares to handle the case one by one?

For example, a caddy module maybe something like this:

func (c PostFormToBody) ServeHTTP(w http.ResponseWriter, r *http.Request, next caddyhttp.Handler) error {
	if r.Method != "POST" && r.Method != "PUT" && r.Method != "PATCH" {
		return next.ServeHTTP(w, r)
	}
	if !strings.Contains(r.Header.Get("Content-Type"), "form-urlencoded") || r.ContentLength == 0 {
		return next.ServeHTTP(w, r)
	}
	if r.PostForm == nil || len(r.PostForm) == 0 {
		return next.ServeHTTP(w, r)
	}
	zerobuf := make([]byte, 0, 0)
	_, err := r.Body.Read(zerobuf)
	if err == io.EOF || err == http.ErrBodyReadAfterClose {
		err = r.Body.Close()
		if err != nil {
			return err
		}
		body := r.PostForm.Encode()
		if r.ContentLength > 0 && int64(len(body)) != r.ContentLength { // When?
			r.ContentLength = int64(len(body))
			r.Header.Set("Content-Length", strconv.Itoa(len(body)))
		}
		r.Body = ioutil.NopCloser(strings.NewReader(body))
	}
	return next.ServeHTTP(w, r) // previous handler reassign request.Body after parse form
}

If this treatment is considered to be general and no side effects，then if the caddy reverse proxy handler can merge it?

4

Interesting… why is the buffer size and capacity 0? Why not use something like ioutil.ReadAll()?

5

why is the buffer size and capacity 0? Why not use something like ioutil.ReadAll()?

What I want to do not really read the request body, just to judge the following case:

  1. the request body length > 0
  2. the previous handler has read the body
  3. the previous handler did not reassign the request body

In such condition, a Read operation should get a io.EOF or http. ErrBodyReadAfterClose according to the net/http implementation here, so I can determine to reassign the request body from PostForm.

if the third is not satisfied, that means the previous handler have reassigned the request body, so here I just make sure the Read operation just return a nil, and not touch the reassigned Body, so there is no need to reassign again, thus the reverse proxy can go through correctly.

Here is a demo:

func TestHTTPRequestBody(t *testing.T) {
	ts := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		zerobuf := make([]byte, 0, 0)
		n, err := r.Body.Read(zerobuf)
		if err != nil {
			t.Fatal(err)
		}
		if n != 0 {
			t.Fatalf("should == 0, got %d", n)
		}
		err = r.ParseForm()
		if err != nil {
			t.Fatal(err)
		}
		w.Write([]byte("request form data: " + r.PostForm.Encode() + "\n"))
		_, err = r.Body.Read(zerobuf)
		if err == io.EOF {
			w.Write([]byte("read body after ParseForm got " + err.Error() + "\n"))
		} else {
			t.Fatal(err)
		}
		err = r.Body.Close()
		if err != nil {
			t.Fatal(err)
		}
		_, err = r.Body.Read(zerobuf)
		if err == http.ErrBodyReadAfterClose {
			w.Write([]byte("read body after Close got " + err.Error() + "\n"))
		} else {
			t.Fatal(err)
		}

		// simulate a previous handler reassign the request body
		r.Body = ioutil.NopCloser(strings.NewReader(r.PostForm.Encode()))
		_, err = r.Body.Read(zerobuf)
		if err != nil {
			t.Fatal(err)
		}
	}))
	defer ts.Close()

	client := httpx.NewRestyV2Client()
	rp, err := client.R().SetFormData(map[string]string{
		"abc": "123",
		"def": "yui",
	}).Post(ts.URL)
	if err != nil {
		t.Fatal(err)
	}
	t.Fatal(rp.String())
}
6

@matt，@francislavoie any comments for this?

7

This topic was automatically closed after 30 days. New replies are no longer allowed.