Sometimes http/2-request hangs up. Need help with debugging

Help
1

1. Caddy version (caddy version):

2.2.1

2. How I run Caddy:

a. System environment:

OS: Manjaro 20.2 Nibia
Kernel: x86_64 Linux 5.4.74-1-MANJARO

b. Command:

`./caddy run`

c. Service/unit/compose file:

paste full file contents here

d. My complete Caddyfile or JSON config:

{
    order http_cache before reverse_proxy
    order header before http_cache
    #order header before reverse_proxy
}

https://cardonecapital.localhost {
    tls internal

    @staticAssets {
        path /img/* /css/* /js/* /fonts/*
    }
    header @staticAssets {
        Cache-Control "public, max-age=31536000, stale-while-revalidate=86400"
        #expires "Wed, 29 Oct 2020 18:35:40 GMT"
        #strict-transport-security "max-age=31536000"
    }

    reverse_proxy {
        header_up host cardonecapital.com
        transport http {
            tls_server_name cardonecapital.com
        }
        to https://cardonecapital.com
    }

    #push /push-trigger.css {
    #    /css/style.min.css
    #}

    # cat reports/https_cardonecapital.localhost_x1_2020-10-31-1525/0001.json | jq '.audits."render-blocking-resources".details.items[].url' | tr -d '"'
    push / {
        /css/bootstrap.min.css
        /css/font-awesome.min.css
        /fonts/fontawesome-webfont.woff2?v=4.7.0
        /css/slick.css
        /css/slick-theme.css
        /css/style.css?v=1.2

        /js/jquery-3.3.1.min.js
        /js/popper.min.js
        /js/bootstrap.min.js
        /js/slick.min.js
        /js/jquery-modal-video.min.js
        /js/custom.js
    }

    http_cache {
        path ./cache
        default_max_age 3h
    }
}

3. The problem I’m having:

I use reverse_proxy, push an my own http_cache module.

Sometimes it stucks at this point in my code:

	log.Printf("%s write content, len=%d, last byte=%d", e.Request.RequestURI, len(content), content[len(content)-1])
	_, err = w.Write(content) // <----- stuck here
	log.Println(e.Request.RequestURI, "writing done")

this code finally execute http2responseWriter.Write (see - The Go Programming Language)

But when I run Delve it shows that some goroutines stuck at writeHeaders or at writeDataFromHandler (see - The Go Programming Language)

It looks like execution stuck at Flush() method, but not Write().

How can I debug this situation?

4. Error messages and/or full log output:

Last part of delve command goroutines output:

  Goroutine 178 - User: /usr/lib/go/src/runtime/sema.go:513 sync.runtime_notifyListWait (0x4831ea)
  Goroutine 194 - User: /home/dionysius/Документы/Исходники/go/github.com/caddyserver/caddy/modules/caddytls/tls.go:397 github.com/caddyserver/caddy/v2/modules/caddytls.(*TLS).keepStorageClean.func1 (0x10fc394)
  Goroutine 195 - User: /usr/lib/go/src/net/http/h2_bundle.go:4428 net/http.(*http2serverConn).serve (0x82e165)
  Goroutine 2953 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2955 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2956 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2957 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2958 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2959 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2960 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2961 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2997 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2998 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 2999 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3000 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3001 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3002 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3028 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3029 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3031 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3032 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3033 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3101 - User: /usr/lib/go/src/net/http/h2_bundle.go:4582 net/http.(*http2serverConn).writeDataFromHandler (0x82f845)
  Goroutine 3102 - User: /usr/lib/go/src/net/http/h2_bundle.go:4582 net/http.(*http2serverConn).writeDataFromHandler (0x82f845)
  Goroutine 3103 - User: /usr/lib/go/src/net/http/h2_bundle.go:4582 net/http.(*http2serverConn).writeDataFromHandler (0x82f845)
  Goroutine 3104 - User: /usr/lib/go/src/net/http/h2_bundle.go:4582 net/http.(*http2serverConn).writeDataFromHandler (0x82f845)
  Goroutine 3105 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3106 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3107 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3109 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3110 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3111 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3112 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3113 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3122 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3124 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3125 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3126 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3127 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3128 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3129 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3130 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3131 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3132 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3133 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3134 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3139 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3140 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3141 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3142 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3143 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3144 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3145 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
  Goroutine 3154 - User: /usr/lib/go/src/net/http/h2_bundle.go:5752 net/http.(*http2serverConn).writeHeaders (0x8394bc)
[172 goroutines]

5. What I already tried:

6. Links to relevant resources:

Maybe this bug is affected this situation: x/net/http2, net/http: client can hang forever if headers' size exceeds connection's buffer size and server hangs past request time · Issue #23559 · golang/go · GitHub

2

out02
out022222×2387 451 KB

3

This topic was automatically closed after 30 days. New replies are no longer allowed.