No worries, sorry it’s been a troublesome time with things.
This topic was automatically closed after 30 days. New replies are no longer allowed.
@LukeHeyburn Did you ever get more information?
Please try upgrading to 2.6.1 (or soon 2.6.2) if you can.
Hey Matt,
I hope you are well.
With the auto-restart script in place I was moved to other tickets on the project. Back on this now though. The issue is still the same as before.
I added the following line to the reboot script but looking at the output I think I may have used the incorrect container name (should be gb-caddy):
timeout 10 docker exec -t caddy wget -q -O - localhost:2019/debug/pprof > pprof.txt
I’ve updated it and ran it manually and it has output. I’ll report back once it goes down again the script auto-runs during the outage.
I could update the caddy version but I’m tempted to find the issue first, unless you think the update will help?
All the best,
Luke
I really do think an upgrade will help. I’ve been working with one of our sponsors to fix a similar issue and a lot has changed between now and the Caddy version you’re using.
Thanks Matt, that’s interesting; I’ll see if I can schedule one in over the weekend.
I have some output now during downtime:
<html>
<head>
<title>/debug/pprof/</title>
<style>
.profile-name{
display:inline-block;
width:6rem;
}
</style>
</head>
<body>
/debug/pprof/<br>
<br>
Types of profiles available:
<table>
<thead><td>Count</td><td>Profile</td></thead>
<tr><td>1392</td><td><a href='allocs?debug=1'>allocs</a></td></tr>
<tr><td>0</td><td><a href='block?debug=1'>block</a></td></tr>
<tr><td>0</td><td><a href='cmdline?debug=1'>cmdline</a></td></tr>
<tr><td>78</td><td><a href='goroutine?debug=1'>goroutine</a></td></tr>
<tr><td>1392</td><td><a href='heap?debug=1'>heap</a></td></tr>
<tr><td>0</td><td><a href='mutex?debug=1'>mutex</a></td></tr>
<tr><td>0</td><td><a href='profile?debug=1'>profile</a></td></tr>
<tr><td>10</td><td><a href='threadcreate?debug=1'>threadcreate</a></td></tr>
<tr><td>0</td><td><a href='trace?debug=1'>trace</a></td></tr>
</table>
<a href="goroutine?debug=2">full goroutine stack dump</a>
<br>
<p>
Profile Descriptions:
<ul>
<li><div class=profile-name>allocs: </div> A sampling of all past memory allocations</li>
<li><div class=profile-name>block: </div> Stack traces that led to blocking on synchronization primitives</li>
<li><div class=profile-name>cmdline: </div> The command line invocation of the current program</li>
<li><div class=profile-name>goroutine: </div> Stack traces of all current goroutines</li>
<li><div class=profile-name>heap: </div> A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample.</li>
<li><div class=profile-name>mutex: </div> Stack traces of holders of contended mutexes</li>
<li><div class=profile-name>profile: </div> CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile.</li>
<li><div class=profile-name>threadcreate: </div> Stack traces that led to the creation of new OS threads</li>
<li><div class=profile-name>trace: </div> A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace.</li>
</ul>
</p>
</body>
</html>
Ok, great. So if you can capture the goroutine dump, that might be helpful!
Also remind me, does this correspond to config reloads?
Also please mind the release notes for any changes that could affect your config, especially experimental or deprecated items.
Also remind me, does this correspond to config reloads?
No, it seems to only happen when the site is busy, hardly ever during the weekend. The config hasn’t changed since we last spoke.
Also please mind the release notes for any changes that could affect your config, especially experimental or deprecated items.
Noted, thank you!
Below is pprof and goroutine output from some more downtime this evening:
<html>
<head>
<title>/debug/pprof/</title>
<style>
.profile-name{
display:inline-block;
width:6rem;
}
</style>
</head>
<body>
/debug/pprof/<br>
<br>
Types of profiles available:
<table>
<thead><td>Count</td><td>Profile</td></thead>
<tr><td>490</td><td><a href='allocs?debug=1'>allocs</a></td></tr>
<tr><td>0</td><td><a href='block?debug=1'>block</a></td></tr>
<tr><td>0</td><td><a href='cmdline?debug=1'>cmdline</a></td></tr>
<tr><td>50</td><td><a href='goroutine?debug=1'>goroutine</a></td></tr>
<tr><td>490</td><td><a href='heap?debug=1'>heap</a></td></tr>
<tr><td>0</td><td><a href='mutex?debug=1'>mutex</a></td></tr>
<tr><td>0</td><td><a href='profile?debug=1'>profile</a></td></tr>
<tr><td>10</td><td><a href='threadcreate?debug=1'>threadcreate</a></td></tr>
<tr><td>0</td><td><a href='trace?debug=1'>trace</a></td></tr>
</table>
<a href="goroutine?debug=2">full goroutine stack dump</a>
<br>
<p>
Profile Descriptions:
<ul>
<li><div class=profile-name>allocs: </div> A sampling of all past memory allocations</li>
<li><div class=profile-name>block: </div> Stack traces that led to blocking on synchronization primitives</li>
<li><div class=profile-name>cmdline: </div> The command line invocation of the current program</li>
<li><div class=profile-name>goroutine: </div> Stack traces of all current goroutines</li>
<li><div class=profile-name>heap: </div> A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample.</li>
<li><div class=profile-name>mutex: </div> Stack traces of holders of contended mutexes</li>
<li><div class=profile-name>profile: </div> CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile.</li>
<li><div class=profile-name>threadcreate: </div> Stack traces that led to the creation of new OS threads</li>
<li><div class=profile-name>trace: </div> A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace.</li>
</ul>
</p>
</body>
</html>
goroutine profile: total 50
6 @ 0x4382d6 0x40640c 0x405e38 0x12acfde 0x12a86bb 0x12a6df8 0xf528df 0xf582db 0xf4924f 0xf58032 0xf4924f 0xf8eae4 0xf528df 0xf582db 0xf4924f 0xf58032 0xf4924f 0xf57fc7 0xf4924f 0xf5dc27 0xf528df 0xf582db 0xf4924f 0xf58032 0xf4924f 0xf5d86d 0xf528df 0xf582db 0xf4924f 0xf58032 0xf4924f 0xf57fc7
# 0x12acfdd github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.handleUpgradeResponse+0xcfd github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/streaming.go:99
# 0x12a86ba github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy+0xb5a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/reverseproxy.go:662
# 0x12a6df7 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP+0xdd7 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/reverseproxy.go:457
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf8eae3 github.com/caddyserver/caddy/v2/modules/caddyhttp/encode.(*Encode).ServeHTTP+0xa3 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/encode/encode.go:130
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf57fc6 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x226 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf5dc26 github.com/caddyserver/caddy/v2/modules/caddyhttp.VarsMiddleware.ServeHTTP+0x246 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/vars.go:53
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf5d86c github.com/caddyserver/caddy/v2/modules/caddyhttp.(*Subroute).ServeHTTP+0x6c github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/subroute.go:74
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf57fc6 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x226 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:203
6 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4efc7a 0x4efc68 0x55c0c9 0x56e685 0x6129bd 0x47d718 0x612ba5 0x610212 0x615aaf 0x615ab0 0x4751b2 0x12ae73c 0x12ae6d9 0x468e21
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4efc79 internal/poll.(*pollDesc).waitRead+0x259 internal/poll/fd_poll_runtime.go:89
# 0x4efc67 internal/poll.(*FD).Read+0x247 internal/poll/fd_unix.go:167
# 0x55c0c8 net.(*netFD).Read+0x28 net/fd_posix.go:56
# 0x56e684 net.(*conn).Read+0x44 net/net.go:183
# 0x6129bc crypto/tls.(*atLeastReader).Read+0x3c crypto/tls/conn.go:777
# 0x47d717 bytes.(*Buffer).ReadFrom+0x97 bytes/buffer.go:204
# 0x612ba4 crypto/tls.(*Conn).readFromUntil+0xe4 crypto/tls/conn.go:799
# 0x610211 crypto/tls.(*Conn).readRecordOrCCS+0x111 crypto/tls/conn.go:606
# 0x615aae crypto/tls.(*Conn).readRecord+0x16e crypto/tls/conn.go:574
# 0x615aaf crypto/tls.(*Conn).Read+0x16f crypto/tls/conn.go:1277
# 0x4751b1 io.copyBuffer+0x1b1 io/io.go:423
# 0x12ae73b io.Copy+0x9b io/io.go:382
# 0x12ae6d8 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.switchProtocolCopier.copyToBackend+0x38 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/streaming.go:272
6 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4efc7a 0x4efc68 0x55c0c9 0x56e685 0x6d9b4f 0x4751b2 0x12ae625 0x12ae5b9 0x468e21
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4efc79 internal/poll.(*pollDesc).waitRead+0x259 internal/poll/fd_poll_runtime.go:89
# 0x4efc67 internal/poll.(*FD).Read+0x247 internal/poll/fd_unix.go:167
# 0x55c0c8 net.(*netFD).Read+0x28 net/fd_posix.go:56
# 0x56e684 net.(*conn).Read+0x44 net/net.go:183
# 0x6d9b4e net/http.(*readWriteCloserBody).Read+0x8e net/http/transport.go:2375
# 0x4751b1 io.copyBuffer+0x1b1 io/io.go:423
# 0x12ae624 io.Copy+0xa4 io/io.go:382
# 0x12ae5b8 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.switchProtocolCopier.copyFromBackend+0x38 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/streaming.go:267
6 @ 0x4382d6 0x4480b2 0x12ad425 0x468e21
# 0x12ad424 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.Handler.handleUpgradeResponse.func1+0xa4 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/streaming.go:61
5 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4efc7a 0x4efc68 0x55c0c9 0x56e685 0x474cfa 0x4e0287 0x4e0288 0x12b54ac 0x12b634d 0x4b3043 0x4b319d 0x64b0ed 0x64aa73 0x12b6926 0x12b7425 0x12b8a4e 0x12a7db8 0x12a6df8 0xf528df 0xf582db 0xf4924f 0xf58032 0xf4924f 0x127b95f 0xf528df 0xf582db 0xf4924f 0xf58032
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4efc79 internal/poll.(*pollDesc).waitRead+0x259 internal/poll/fd_poll_runtime.go:89
# 0x4efc67 internal/poll.(*FD).Read+0x247 internal/poll/fd_unix.go:167
# 0x55c0c8 net.(*netFD).Read+0x28 net/fd_posix.go:56
# 0x56e684 net.(*conn).Read+0x44 net/net.go:183
# 0x474cf9 io.ReadAtLeast+0x99 io/io.go:328
# 0x4e0286 io.ReadFull+0xc46 io/io.go:347
# 0x4e0287 encoding/binary.Read+0xc47 encoding/binary/binary.go:256
# 0x12b54ab github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*record).read+0x4b github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:151
# 0x12b634c github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*streamReader).Read+0x18c github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:351
# 0x4b3042 bufio.(*Reader).fill+0x102 bufio/bufio.go:101
# 0x4b319c bufio.(*Reader).Peek+0x5c bufio/bufio.go:139
# 0x64b0ec net/textproto.(*Reader).upcomingHeaderNewlines+0x2c net/textproto/reader.go:568
# 0x64aa72 net/textproto.(*Reader).ReadMIMEHeader+0x32 net/textproto/reader.go:488
# 0x12b6925 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*FCGIClient).Request+0x205 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:421
# 0x12b7424 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*FCGIClient).Post+0x3c4 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:500
# 0x12b8a4d github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.Transport.RoundTrip+0x9ed github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go:158
# 0x12a7db7 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy+0x257 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/reverseproxy.go:573
# 0x12a6df7 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP+0xdd7 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/reverseproxy.go:457
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0x127b95e github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite.Rewrite.ServeHTTP+0x3fe github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/rewrite/rewrite.go:118
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
2 @ 0x4382d6 0x40640c 0x405e78 0x1302ce5 0x468e21
# 0x1302ce4 gopkg.in/natefinch/lumberjack%2ev2.(*Logger).millRun+0x44 gopkg.in/natefinch/lumberjack.v2@v2.0.0/lumberjack.go:379
2 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4efc7a 0x4efc68 0x55c0c9 0x56e685 0x474cfa 0x4e0287 0x4e0288 0x12b54ac 0x12b634d 0x4b3043 0x4b319d 0x64b0ed 0x64aa73 0x12b6926 0x12b6e0e 0x12b8948 0x12a7db8 0x12a6df8 0xf528df 0xf582db 0xf4924f 0xf58032 0xf4924f 0x127b95f 0xf528df 0xf582db 0xf4924f 0xf58032
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4efc79 internal/poll.(*pollDesc).waitRead+0x259 internal/poll/fd_poll_runtime.go:89
# 0x4efc67 internal/poll.(*FD).Read+0x247 internal/poll/fd_unix.go:167
# 0x55c0c8 net.(*netFD).Read+0x28 net/fd_posix.go:56
# 0x56e684 net.(*conn).Read+0x44 net/net.go:183
# 0x474cf9 io.ReadAtLeast+0x99 io/io.go:328
# 0x4e0286 io.ReadFull+0xc46 io/io.go:347
# 0x4e0287 encoding/binary.Read+0xc47 encoding/binary/binary.go:256
# 0x12b54ab github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*record).read+0x4b github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:151
# 0x12b634c github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*streamReader).Read+0x18c github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:351
# 0x4b3042 bufio.(*Reader).fill+0x102 bufio/bufio.go:101
# 0x4b319c bufio.(*Reader).Peek+0x5c bufio/bufio.go:139
# 0x64b0ec net/textproto.(*Reader).upcomingHeaderNewlines+0x2c net/textproto/reader.go:568
# 0x64aa72 net/textproto.(*Reader).ReadMIMEHeader+0x32 net/textproto/reader.go:488
# 0x12b6925 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*FCGIClient).Request+0x205 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:421
# 0x12b6e0d github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.(*FCGIClient).Get+0xed github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/client.go:459
# 0x12b8947 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy/fastcgi.Transport.RoundTrip+0x8e7 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/fastcgi/fastcgi.go:154
# 0x12a7db7 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).reverseProxy+0x257 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/reverseproxy.go:573
# 0x12a6df7 github.com/caddyserver/caddy/v2/modules/caddyhttp/reverseproxy.(*Handler).ServeHTTP+0xdd7 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/reverseproxy/reverseproxy.go:457
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0x127b95e github.com/caddyserver/caddy/v2/modules/caddyhttp/rewrite.Rewrite.ServeHTTP+0x3fe github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/rewrite/rewrite.go:118
# 0xf528de github.com/caddyserver/caddy/v2/modules/caddyhttp.(*metricsInstrumentedHandler).ServeHTTP+0x51e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/metrics.go:133
# 0xf582da github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapMiddleware.func1.1+0x3a github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:263
# 0xf4924e github.com/caddyserver/caddy/v2/modules/caddyhttp.HandlerFunc.ServeHTTP+0x2e github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/caddyhttp.go:57
# 0xf58031 github.com/caddyserver/caddy/v2/modules/caddyhttp.wrapRoute.func1.1+0x291 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddyhttp/routes.go:235
2 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4efc7a 0x4efc68 0x55c0c9 0x56e685 0x6129bd 0x47d718 0x612ba5 0x610212 0x615aaf 0x615ab0 0x474cfa 0x68564e 0x685608 0x685d95 0x690331 0x468e21
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4efc79 internal/poll.(*pollDesc).waitRead+0x259 internal/poll/fd_poll_runtime.go:89
# 0x4efc67 internal/poll.(*FD).Read+0x247 internal/poll/fd_unix.go:167
# 0x55c0c8 net.(*netFD).Read+0x28 net/fd_posix.go:56
# 0x56e684 net.(*conn).Read+0x44 net/net.go:183
# 0x6129bc crypto/tls.(*atLeastReader).Read+0x3c crypto/tls/conn.go:777
# 0x47d717 bytes.(*Buffer).ReadFrom+0x97 bytes/buffer.go:204
# 0x612ba4 crypto/tls.(*Conn).readFromUntil+0xe4 crypto/tls/conn.go:799
# 0x610211 crypto/tls.(*Conn).readRecordOrCCS+0x111 crypto/tls/conn.go:606
# 0x615aae crypto/tls.(*Conn).readRecord+0x16e crypto/tls/conn.go:574
# 0x615aaf crypto/tls.(*Conn).Read+0x16f crypto/tls/conn.go:1277
# 0x474cf9 io.ReadAtLeast+0x99 io/io.go:328
# 0x68564d io.ReadFull+0x6d io/io.go:347
# 0x685607 net/http.http2readFrameHeader+0x27 net/http/h2_bundle.go:1553
# 0x685d94 net/http.(*http2Framer).ReadFrame+0x94 net/http/h2_bundle.go:1811
# 0x690330 net/http.(*http2serverConn).readFrames+0x90 net/http/h2_bundle.go:4398
2 @ 0x4382d6 0x4480b2 0x69124d 0x68f5b1 0x68eae6 0x6bd878 0x468e21
# 0x69124c net/http.(*http2serverConn).serve+0x8ec net/http/h2_bundle.go:4512
# 0x68f5b0 net/http.(*http2Server).ServeConn+0xa70 net/http/h2_bundle.go:4122
# 0x68eae5 net/http.http2ConfigureServer.func1+0xe5 net/http/h2_bundle.go:3948
# 0x6bd877 net/http.(*conn).serve+0x6b7 net/http/server.go:1838
1 @ 0x40b974 0x465498 0x90fb99 0x468e21
# 0x465497 os/signal.signal_recv+0x97 runtime/sigqueue.go:169
# 0x90fb98 os/signal.loop+0x18 os/signal/signal_unix.go:24
1 @ 0x4382d6 0x40640c 0x405e38 0x92ba5f 0x468e21
# 0x92ba5e github.com/caddyserver/caddy/v2.trapSignalsCrossPlatform.func1+0xde github.com/caddyserver/caddy/v2@v2.4.5/sigtrap.go:42
1 @ 0x4382d6 0x40640c 0x405e78 0x92bdc5 0x468e21
# 0x92bdc4 github.com/caddyserver/caddy/v2.trapSignalsPosix.func1+0x104 github.com/caddyserver/caddy/v2@v2.4.5/sigtrap_posix.go:34
1 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4efc7a 0x4efc68 0x55c0c9 0x56e685 0x6b7e7f 0x468e21
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4efc79 internal/poll.(*pollDesc).waitRead+0x259 internal/poll/fd_poll_runtime.go:89
# 0x4efc67 internal/poll.(*FD).Read+0x247 internal/poll/fd_unix.go:167
# 0x55c0c8 net.(*netFD).Read+0x28 net/fd_posix.go:56
# 0x56e684 net.(*conn).Read+0x44 net/net.go:183
# 0x6b7e7e net/http.(*connReader).backgroundRead+0x3e net/http/server.go:672
1 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4f1c6c 0x4f1c59 0x55d875 0x5777a8 0x57697d 0x9218df 0x645a8d 0x6c26d4 0x468e21
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4f1c6b internal/poll.(*pollDesc).waitRead+0x22b internal/poll/fd_poll_runtime.go:89
# 0x4f1c58 internal/poll.(*FD).Accept+0x218 internal/poll/fd_unix.go:402
# 0x55d874 net.(*netFD).accept+0x34 net/fd_unix.go:173
# 0x5777a7 net.(*TCPListener).accept+0x27 net/tcpsock_posix.go:140
# 0x57697c net.(*TCPListener).Accept+0x3c net/tcpsock.go:262
# 0x9218de github.com/caddyserver/caddy/v2.(*fakeCloseListener).Accept+0x3e github.com/caddyserver/caddy/v2@v2.4.5/listeners.go:122
# 0x645a8c crypto/tls.(*listener).Accept+0x2c crypto/tls/tls.go:66
# 0x6c26d3 net/http.(*Server).Serve+0x393 net/http/server.go:3001
1 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4f1c6c 0x4f1c59 0x55d875 0x5777a8 0x57697d 0x9218df 0x6c26d4 0x468e21
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4f1c6b internal/poll.(*pollDesc).waitRead+0x22b internal/poll/fd_poll_runtime.go:89
# 0x4f1c58 internal/poll.(*FD).Accept+0x218 internal/poll/fd_unix.go:402
# 0x55d874 net.(*netFD).accept+0x34 net/fd_unix.go:173
# 0x5777a7 net.(*TCPListener).accept+0x27 net/tcpsock_posix.go:140
# 0x57697c net.(*TCPListener).Accept+0x3c net/tcpsock.go:262
# 0x9218de github.com/caddyserver/caddy/v2.(*fakeCloseListener).Accept+0x3e github.com/caddyserver/caddy/v2@v2.4.5/listeners.go:122
# 0x6c26d3 net/http.(*Server).Serve+0x393 net/http/server.go:3001
1 @ 0x4382d6 0x4309de 0x4635e9 0x4ee912 0x4f1c6c 0x4f1c59 0x55d875 0x5777a8 0x57697d 0x9218df 0x6c26d4 0x911c5e 0x468e21
# 0x4635e8 internal/poll.runtime_pollWait+0x88 runtime/netpoll.go:229
# 0x4ee911 internal/poll.(*pollDesc).wait+0x31 internal/poll/fd_poll_runtime.go:84
# 0x4f1c6b internal/poll.(*pollDesc).waitRead+0x22b internal/poll/fd_poll_runtime.go:89
# 0x4f1c58 internal/poll.(*FD).Accept+0x218 internal/poll/fd_unix.go:402
# 0x55d874 net.(*netFD).accept+0x34 net/fd_unix.go:173
# 0x5777a7 net.(*TCPListener).accept+0x27 net/tcpsock_posix.go:140
# 0x57697c net.(*TCPListener).Accept+0x3c net/tcpsock.go:262
# 0x9218de github.com/caddyserver/caddy/v2.(*fakeCloseListener).Accept+0x3e github.com/caddyserver/caddy/v2@v2.4.5/listeners.go:122
# 0x6c26d3 net/http.(*Server).Serve+0x393 net/http/server.go:3001
# 0x911c5d github.com/caddyserver/caddy/v2.replaceLocalAdminServer.func2+0xbd github.com/caddyserver/caddy/v2@v2.4.5/admin.go:354
1 @ 0x4382d6 0x44792c 0x93d538 0x942062 0x13328d7 0x437f07 0x468e21
# 0x93d537 github.com/caddyserver/caddy/v2/cmd.cmdRun+0xb77 github.com/caddyserver/caddy/v2@v2.4.5/cmd/commandfuncs.go:275
# 0x942061 github.com/caddyserver/caddy/v2/cmd.Main+0x401 github.com/caddyserver/caddy/v2@v2.4.5/cmd/main.go:85
# 0x13328d6 main.main+0x16 ./main.go:37
# 0x437f06 runtime.main+0x226 runtime/proc.go:255
1 @ 0x4382d6 0x4480b2 0x81250c 0x468e21
# 0x81250b github.com/caddyserver/certmagic.(*Cache).maintainAssets+0x30b github.com/caddyserver/certmagic@v0.14.5/maintain.go:71
1 @ 0x4382d6 0x4480b2 0x81b055 0x81a891 0x468e21
# 0x81b054 github.com/caddyserver/certmagic.(*RingBufferRateLimiter).permit+0x94 github.com/caddyserver/certmagic@v0.14.5/ratelimiter.go:216
# 0x81a890 github.com/caddyserver/certmagic.(*RingBufferRateLimiter).loop+0x90 github.com/caddyserver/certmagic@v0.14.5/ratelimiter.go:89
1 @ 0x4382d6 0x4480b2 0x81b055 0x81a9bd 0x468e21
# 0x81b054 github.com/caddyserver/certmagic.(*RingBufferRateLimiter).permit+0x94 github.com/caddyserver/certmagic@v0.14.5/ratelimiter.go:216
# 0x81a9bc github.com/caddyserver/certmagic.(*RingBufferRateLimiter).loop+0x1bc github.com/caddyserver/certmagic@v0.14.5/ratelimiter.go:103
1 @ 0x4382d6 0x4480b2 0xd22805 0x468e21
# 0xd22804 github.com/caddyserver/caddy/v2/modules/caddytls.(*TLS).keepStorageClean.func1+0xa4 github.com/caddyserver/caddy/v2@v2.4.5/modules/caddytls/tls.go:446
1 @ 0x463105 0x730bd5 0x7309ed 0x72db6b 0x73c77a 0x73d32e 0x6bebef 0x926cfa 0x6bebef 0x6c04e9 0x914ef2 0x914938 0x6c215b 0x6bdcc8 0x468e21
# 0x463104 runtime/pprof.runtime_goroutineProfileWithLabels+0x24 runtime/mprof.go:746
# 0x730bd4 runtime/pprof.writeRuntimeProfile+0xb4 runtime/pprof/pprof.go:724
# 0x7309ec runtime/pprof.writeGoroutine+0x4c runtime/pprof/pprof.go:684
# 0x72db6a runtime/pprof.(*Profile).WriteTo+0x14a runtime/pprof/pprof.go:331
# 0x73c779 net/http/pprof.handler.ServeHTTP+0x499 net/http/pprof/pprof.go:253
# 0x73d32d net/http/pprof.Index+0x12d net/http/pprof/pprof.go:371
# 0x6bebee net/http.HandlerFunc.ServeHTTP+0x2e net/http/server.go:2046
# 0x926cf9 github.com/caddyserver/caddy/v2.instrumentHandlerCounter.func1+0xb9 github.com/caddyserver/caddy/v2@v2.4.5/metrics.go:47
# 0x6bebee net/http.HandlerFunc.ServeHTTP+0x2e net/http/server.go:2046
# 0x6c04e8 net/http.(*ServeMux).ServeHTTP+0x148 net/http/server.go:2424
# 0x914ef1 github.com/caddyserver/caddy/v2.adminHandler.serveHTTP+0x531 github.com/caddyserver/caddy/v2@v2.4.5/admin.go:729
# 0x914937 github.com/caddyserver/caddy/v2.adminHandler.ServeHTTP+0x717 github.com/caddyserver/caddy/v2@v2.4.5/admin.go:681
# 0x6c215a net/http.serverHandler.ServeHTTP+0x43a net/http/server.go:2878
# 0x6bdcc7 net/http.(*conn).serve+0xb07 net/http/server.go:1929
1 Like
Thank you. That’s very interesting – I’d be exceedingly interested in the result after upgrading to v2.6.2. We made a lot of fixes that could be relevant:
- Simpler, more reliable listener reuse mechanism
- Better closing of websocket connections on both ends if one side closes
- Improved streaming logic in the proxy
- Several significant performance increases
I would also ask for full logs (enable debug mode – debug in the global options, and enable access logs with the log directive in your site block, if you haven’t already) around the time when it goes down (before, right at that time, and after), this will help complete the picture, hopefully. If you have those even from your 2.4 deployment, that’d be helpful.
If it doesn’t reveal anything useful, there’s a good chance it’s something else in the stack (network, kernel, infrastructure, etc) and not necessarily Caddy code.
This topic was automatically closed after 28 days. New replies are no longer allowed.
Hi Matt,
I ran the update just over a week ago and unfortunately the issue persists. Last week the site went down 4 times.
5 times the week prior, 18 prior to that and 5 times 3 weeks ago.
However, whilst working locally I did actually experience what appears to be the same error where the php container did not respond to any more queries. I had WS running through one php container and a separate container for “cron” tasks which is also a php container which both appeared to continue working without issue. All three of these containers use the same codebase and database. It was the only the web serving container which was failing to respond. However, you may be glad to know that we do not run caddy locally, instead opting for the nginx:alpine image as none of the https bits are required.
It seems to be pointing back to an issue with the php container setup instead, if it is indeed the same issue. I’m going to enable slowlogs in the php container next and see if that records any pattern. I’ll report back if I can confirm that this is the issue. I have not done much debugging at this level with php though so I have some more reading to do!
Thank you for all of your advice so far.
3 Likes
Great, thanks for the update. I’m exceedingly interested in this, so let me know what you find out! I’m sure others in the future who search for this problem will also find your results relevant.
This topic was automatically closed after 26 days. New replies are no longer allowed.