1. Caddy version (caddy version):
v2.4.3 h1:Y1FaV2N4WO3rBqxSYA8UZsZTQdN+PwcoOcAiZTM8C0I=
2. How I run Caddy:
Caddy is run on 2 servers both connected to redis + Linode object storage for shared storage using caddy.storage.s3 module
a. System environment:
Ubuntu 20.0.4 LTS, systemd
b. Command:
systemctl start caddy
c. Service/unit/compose file:
[Unit]
Description=Caddy
Documentation=https://caddyserver.com/docs/
After=network.target network-online.target
Requires=network-online.target
[Service]
Type=notify
User=caddy
Group=caddy
ExecStart=/usr/bin/caddy run --environ --config /etc/caddy/config.json
ExecReload=/usr/bin/caddy reload --config /etc/caddy/config.json
TimeoutStartSec=1200s
TimeoutStopSec=10s
Restart=always
RestartSec=3
LimitNOFILE=1048576
LimitNPROC=512
PrivateTmp=true
ProtectSystem=full
AmbientCapabilities=CAP_NET_BIND_SERVICE
[Install]
WantedBy=multi-user.target
3. The problem I’m having:
Caddy renewing certificates and crashed, seems to be same as this
4. Error messages and/or full log output:
Jul 14 05:12:59 venus caddy[833]: {“level”:“info”,“ts”:1626239579.7318494,“logger”:“tls.renew”,“msg”:“lock acquired”,“identifier”:“api.theembassy.staffconnect-app.com”}
Jul 14 05:12:59 venus caddy[833]: {“level”:“info”,“ts”:1626239579.7318974,“logger”:“caddy.storage.s3”,“msg”:"Load: certificates/acme-v02.api.letsencrypt.org-directory/api.theembassy.staffconnect-app.com/api.theembassy.staffconnect-app.co>
Jul 14 05:12:59 venus caddy[833]: fatal error: concurrent map writes
Jul 14 05:12:59 venus caddy[833]: goroutine 2341366 [running]:
Jul 14 05:12:59 venus caddy[833]: runtime.throw(0x1a597ec, 0x15)
Jul 14 05:12:59 venus caddy[833]: runtime/panic.go:1117 +0x72 fp=0xc002a176e8 sp=0xc002a176b8 pc=0x438652
Jul 14 05:12:59 venus caddy[833]: runtime.mapassign_faststr(0x17ecde0, 0xc00038c390, 0xc000740cf0, 0x30, 0x30)
Jul 14 05:12:59 venus caddy[833]: runtime/map_faststr.go:211 +0x3f1 fp=0xc002a17750 sp=0xc002a176e8 pc=0x414911
Jul 14 05:12:59 venus caddy[833]: github.com/ss098/certmagic-s3.(*S3).Lock(0xc0000b2630, 0x1d500d0, 0xc0002f55c0, 0xc000740cf0, 0x30, 0x2, 0x30)
Jul 14 05:12:59 venus caddy[833]: github.com/ss098/certmagic-s3@v0.0.0-20210602125901-e56543144a58/s3.go:108 +0x1f6 fp=0xc002a177f8 sp=0xc002a17750 pc=0x16a1716
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.acquireLock(0x1d500d0, 0xc0002f55c0, 0x1d5f478, 0xc0000b2630, 0xc000740cf0, 0x30, 0x30, 0x27fc840)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/storage.go:236 +0x62 fp=0xc002a17850 sp=0xc002a177f8 pc=0x8b1e22
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Config).renewCert(0xc0004de5a0, 0x1d500d0, 0xc0002f55c0, 0xc001612f90, 0x25, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/config.go:662 +0x25e fp=0xc002a17a80 sp=0xc002a17850 pc=0x8919fe
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Config).RenewCertAsync(…)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/config.go:639
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Cache).queueRenewalTask.func1(0xc003585018, 0xc00086f180)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/maintain.go:254 +0x1ba fp=0xc002a17ea0 sp=0xc002a17a80 pc=0x8b80da
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*jobManager).worker(0x27ffb40)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/async.go:73 +0x103 fp=0xc002a17fd8 sp=0xc002a17ea0 pc=0x888083
Jul 14 05:12:59 venus caddy[833]: runtime.goexit()
Jul 14 05:12:59 venus caddy[833]: runtime/asm_amd64.s:1371 +0x1 fp=0xc002a17fe0 sp=0xc002a17fd8 pc=0x472841
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/certmagic.(*jobManager).Submit
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/async.go:50 +0x12d
Jul 14 05:12:59 venus caddy[833]: goroutine 1 [select (no cases), 1278 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2/cmd.cmdRun(0xc0001057a0, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/cmd/commandfuncs.go:276 +0x1395
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2/cmd.Main()
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/cmd/main.go:85 +0x25b
Jul 14 05:12:59 venus caddy[833]: main.main()
Jul 14 05:12:59 venus caddy[833]: caddy/main.go:12 +0x25
Jul 14 05:12:59 venus caddy[833]: goroutine 17 [select, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*RingBufferRateLimiter).permit(0xc00009c190)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/ratelimiter.go:216 +0xb2
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*RingBufferRateLimiter).loop(0xc00009c190)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/ratelimiter.go:89 +0xa8
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/certmagic.NewRateLimiter
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/ratelimiter.go:45 +0x148
Jul 14 05:12:59 venus caddy[833]: goroutine 13 [chan receive, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.trapSignalsCrossPlatform.func1()
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/sigtrap.go:42 +0x129
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/caddy/v2.trapSignalsCrossPlatform
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/sigtrap.go:37 +0x35
Jul 14 05:12:59 venus caddy[833]: goroutine 14 [chan receive, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.trapSignalsPosix.func1()
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/sigtrap_posix.go:34 +0x139
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/caddy/v2.trapSignalsPosix
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/sigtrap_posix.go:30 +0x35
Jul 14 05:12:59 venus caddy[833]: goroutine 19 [syscall, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: os/signal.signal_recv(0x0)
Jul 14 05:12:59 venus caddy[833]: runtime/sigqueue.go:168 +0xa5
Jul 14 05:12:59 venus caddy[833]: os/signal.loop()
Jul 14 05:12:59 venus caddy[833]: os/signal/signal_unix.go:23 +0x25
Jul 14 05:12:59 venus caddy[833]: created by os/signal.Notify.func1.1
Jul 14 05:12:59 venus caddy[833]: os/signal/signal.go:151 +0x45
Jul 14 05:12:59 venus caddy[833]: goroutine 67 [IO wait, 1280 minutes]:
Jul 14 05:12:59 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15eace7d8, 0x72, 0x0)
Jul 14 05:12:59 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc000470c98, 0x72, 0x0, 0x0, 0x1a12a0f)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*FD).Accept(0xc000470c80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_unix.go:401 +0x212
Jul 14 05:12:59 venus caddy[833]: net.(*netFD).accept(0xc000470c80, 0x50, 0xc0000f1cd0, 0x40de72)
Jul 14 05:12:59 venus caddy[833]: net/fd_unix.go:172 +0x45
Jul 14 05:12:59 venus caddy[833]: net.(*TCPListener).accept(0xc0006c4138, 0xc000484820, 0x7fe15e7d3768, 0x30)
Jul 14 05:12:59 venus caddy[833]: net/tcpsock_posix.go:139 +0x32
Jul 14 05:12:59 venus caddy[833]: net.(*TCPListener).Accept(0xc0006c4138, 0x282c1a0, 0xc000382d80, 0xc0000f1d00, 0x40e398)
Jul 14 05:12:59 venus caddy[833]: net/tcpsock.go:261 +0x65
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.(*fakeCloseListener).Accept(0xc0006c0680, 0x1902040, 0xc0006c2d50, 0x17dab00, 0x27f59c0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/listeners.go:122 +0x43
Jul 14 05:12:59 venus caddy[833]: net/http.(*Server).Serve(0xc00036a2a0, 0x1d48f90, 0xc0006c0680, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: net/http/server.go:2981 +0x285
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2.replaceLocalAdminServer.func2(0x1d48f90, 0xc0006c0680, 0xc0004847d0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/admin.go:343 +0x5c
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/caddy/v2.replaceLocalAdminServer
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/caddy/v2@v2.4.3/admin.go:342 +0x455
Jul 14 05:12:59 venus caddy[833]: goroutine 68 [select]:
Jul 14 05:12:59 venus caddy[833]: github.com/go-redis/redis/v8/internal/pool.(*ConnPool).reaper(0xc0004de0a0, 0xdf8475800)
Jul 14 05:12:59 venus caddy[833]: github.com/go-redis/redis/v8@v8.9.0/internal/pool/pool.go:455 +0xd7
Jul 14 05:12:59 venus caddy[833]: created by github.com/go-redis/redis/v8/internal/pool.NewConnPool
Jul 14 05:12:59 venus caddy[833]: github.com/go-redis/redis/v8@v8.9.0/internal/pool/pool.go:110 +0x1fc
Jul 14 05:12:59 venus caddy[833]: goroutine 69 [select]:
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic.(*Cache).maintainAssets(0xc000188380, 0x0)
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/maintain.go:71 +0x1ab
Jul 14 05:12:59 venus caddy[833]: created by github.com/caddyserver/certmagic.NewCache
Jul 14 05:12:59 venus caddy[833]: github.com/caddyserver/certmagic@v0.14.0/cache.go:121 +0x172
Jul 14 05:12:59 venus caddy[833]: goroutine 77 [IO wait, 1 minutes]:
Jul 14 05:12:59 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15eace608, 0x72, 0x0)
Jul 14 05:12:59 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc00016b098, 0x72, 0x0, 0x0, 0x1a12a0f)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*FD).Accept(0xc00016b080, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_unix.go:401 +0x212
Jul 14 05:12:59 venus caddy[833]: net.(*netFD).accept(0xc00016b080, 0x203000, 0x203000, 0x0)
Jul 14 05:12:59 venus caddy[833]: net/fd_unix.go:172 +0x45
similar lines removed for brevity
Jul 14 05:12:59 venus caddy[833]: goroutine 1161462 [IO wait, 527 minutes]:
Jul 14 05:12:59 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15e5a5fc8, 0x72, 0xffffffffffffffff)
Jul 14 05:12:59 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc004545998, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:12:59 venus caddy[833]: internal/poll.(*FD).Read(0xc004545980, 0xc000f72000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 14 05:12:59 venus caddy[833]: internal/poll/fd_unix.go:166 +0x1d5
Jul 14 05:13:00 venus caddy[833]: net.(*netFD).Read(0xc004545980, 0xc000f72000, 0x1000, 0x1000, 0x101, 0xc001324a80, 0xc002a9e6a0)
Jul 14 05:13:00 venus caddy[833]: net/fd_posix.go:55 +0x4f
Jul 14 05:13:00 venus caddy[833]: net.(*conn).Read(0xc0009a80c0, 0xc000f72000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/net.go:183 +0x91
Jul 14 05:13:00 venus caddy[833]: net/http.(*connReader).Read(0xc003e60510, 0xc000f72000, 0x1000, 0x1000, 0x1ba, 0x4149a8, 0xc002a9e7c8)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:800 +0x1b9
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).fill(0xc003b04e40)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:101 +0x108
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).ReadSlice(0xc003b04e40, 0x418e0a, 0x7fe15e8292d8, 0x0, 0xc003e60090, 0x1a124ac, 0x100000dd2)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:360 +0x3d
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).ReadLine(0xc003b04e40, 0x203000, 0x203000, 0x756ea1fb2446cc, 0x203001, 0x203001, 0x203001)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:389 +0x34
Jul 14 05:13:00 venus caddy[833]: net/textproto.(*Reader).readLineSlice(0xc004c4c330, 0xc0034c9800, 0x15, 0x7fe185793108, 0x18, 0xc005e74b28)
Jul 14 05:13:00 venus caddy[833]: net/textproto/reader.go:57 +0xd6
Jul 14 05:13:00 venus caddy[833]: net/textproto.(*Reader).ReadLine(…)
Jul 14 05:13:00 venus caddy[833]: net/textproto/reader.go:38
Jul 14 05:13:00 venus caddy[833]: net/http.readRequest(0xc003b04e40, 0x0, 0xc0034c9800, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/http/request.go:1027 +0xaa
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).readRequest(0xc000add860, 0x1d500d0, 0xc001324a40, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:986 +0x19d
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc000add860, 0x1d50178, 0xc001324a40)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1878 +0x705
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus caddy[833]: goroutine 2492 [select, 1279 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc0015e5920, 0xc0015e5980, 0xc0015e59e0, 0xc00158b920, 0xc0019a8dc0, 0xc0019ae210, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7@v7.0.10/api-get-object.go:74 +0x148
Jul 14 05:13:00 venus caddy[833]: created by github.com/minio/minio-go/v7.Client.GetObject
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7@v7.0.10/api-get-object.go:65 +0x397
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: goroutine 2340058 [select, 1 minutes]:
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2serverConn).serve(0xc001e80780)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4428 +0x579
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2Server).ServeConn(0xc0000c9d80, 0x1d5d4d8, 0xc00171c000, 0xc000ec3b68)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4038 +0x725
Jul 14 05:13:00 venus caddy[833]: net/http.http2ConfigureServer.func1(0xc00036a1c0, 0xc00171c000, 0x1d33800, 0xc002be7540)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:3864 +0xf0
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc001e7a5a0, 0x1d50178, 0xc0003cacc0)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1861 +0x1535
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus caddy[833]: goroutine 2956 [select, 1279 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc000a988a0, 0xc000a98900, 0xc000a98a80, 0xc0017fde40, 0xc001149140, 0xc0019aedc0, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7@v7.0.10/api-get-object.go:74 +0x148
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: goroutine 2340626 [IO wait, 1 minutes]:
Jul 14 05:13:00 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15e5f3d78, 0x72, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc0038fc798, 0x72, 0x200, 0x205, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*FD).Read(0xc0038fc780, 0xc002c72900, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_unix.go:166 +0x1d5
Jul 14 05:13:00 venus caddy[833]: net.(*netFD).Read(0xc0038fc780, 0xc002c72900, 0x205, 0x205, 0x200, 0xc002c72900, 0x5)
Jul 14 05:13:00 venus caddy[833]: net/fd_posix.go:55 +0x4f
Jul 14 05:13:00 venus caddy[833]: net.(*conn).Read(0xc0009a80e0, 0xc002c72900, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/net.go:183 +0x91
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*atLeastReader).Read(0xc0006ce288, 0xc002c72900, 0x205, 0x205, 0x200, 0xc0020a0000, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:776 +0x63
Jul 14 05:13:00 venus caddy[833]: bytes.(*Buffer).ReadFrom(0xc000b09778, 0x1d2d7c0, 0xc0006ce288, 0x40b9c5, 0x1818a00, 0x198d260)
Jul 14 05:13:00 venus caddy[833]: bytes/buffer.go:204 +0xbe
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readFromUntil(0xc000b09500, 0x1d31080, 0xc0009a80e0, 0x5, 0xc0009a80e0, 0x11)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:798 +0xf3
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecordOrCCS(0xc000b09500, 0x0, 0x0, 0xc00323dd38)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:605 +0x115
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecord(…)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:573
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).Read(0xc000b09500, 0xc0015da1f8, 0x9, 0x9, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:1276 +0x165
Jul 14 05:13:00 venus caddy[833]: io.ReadAtLeast(0x7fe15e6c89a0, 0xc000b09500, 0xc0015da1f8, 0x9, 0x9, 0x9, 0x101006dccc5, 0x0, 0x2)
Jul 14 05:13:00 venus caddy[833]: io/io.go:328 +0x87
Jul 14 05:13:00 venus caddy[833]: io.ReadFull(…)
Jul 14 05:13:00 venus caddy[833]: io/io.go:347
Jul 14 05:13:00 venus caddy[833]: net/http.http2readFrameHeader(0xc0015da1f8, 0x9, 0x9, 0x7fe15e6c89a0, 0xc000b09500, 0x0, 0xc000000000, 0xc0016daf00, 0xc00323df70)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1477 +0x89
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2Framer).ReadFrame(0xc0015da1c0, 0xc00323df38, 0x0, 0x0, 0x2)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1735 +0xa5
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2serverConn).readFrames(0xc001e80900)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4314 +0xa9
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*http2serverConn).serve
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4420 +0x366
Jul 14 05:13:00 venus caddy[833]: goroutine 3073 [select, 1279 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc001832720, 0xc001832780, 0xc0018327e0, 0xc00172da90, 0xc001610c40, 0xc00189c0b0, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7@v7.0.10/api-get-object.go:74 +0x148
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: goroutine 2341476 [select]:
Jul 14 05:13:00 venus caddy[833]: net/http.(*persistConn).writeLoop(0xc00066b440)
Jul 14 05:13:00 venus caddy[833]: net/http/transport.go:2382 +0xf7
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Transport).dialConn
Jul 14 05:13:00 venus caddy[833]: net/http/transport.go:1744 +0xc9c
Jul 14 05:13:00 venus caddy[833]: goroutine 3219 [select, 1278 minutes]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc00144fbc0, 0xc00144fc20, 0xc00144fc80, 0xc0015a8c60, 0xc0018a1b80, 0xc00144c210, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7@v7.0.10/api-get-object.go:74 +0x148
similar lines removed for brevity
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc002493400, 0x1d50178, 0xc001bf1880)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1977 +0xa47
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus caddy[833]: io/io.go:328 +0x87
Jul 14 05:13:00 venus caddy[833]: io.ReadFull(…)
Jul 14 05:13:00 venus caddy[833]: io/io.go:347
Jul 14 05:13:00 venus caddy[833]: net/http.http2readFrameHeader(0xc001f21df8, 0x9, 0x9, 0x7fe15e6c89a0, 0xc002f54380, 0x0, 0xc000000000, 0xc0050d4300, 0xc000cdaf70)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1477 +0x89
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2Framer).ReadFrame(0xc001f21dc0, 0xc000cdaf38, 0x0, 0x0, 0x2)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:1735 +0xa5
Jul 14 05:13:00 venus caddy[833]: net/http.(*http2serverConn).readFrames(0xc003d28780)
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4314 +0xa9
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*http2serverConn).serve
Jul 14 05:13:00 venus caddy[833]: net/http/h2_bundle.go:4420 +0x366
Jul 14 05:13:00 venus caddy[833]: goroutine 2341491 [select]:
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7.Client.GetObject.func1(0xc003bf2300, 0xc003bf23c0, 0xc003bf2420, 0xc003df8360, 0xc001e7d340, 0xc001e54210, 0x1d50108, 0xc00003e050, 0xc0001fb950, 0xa, …)
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7@v7.0.10/api-get-object.go:74 +0x148
Jul 14 05:13:00 venus caddy[833]: created by github.com/minio/minio-go/v7.Client.GetObject
Jul 14 05:13:00 venus caddy[833]: github.com/minio/minio-go/v7@v7.0.10/api-get-object.go:65 +0x397
Jul 14 05:13:00 venus caddy[833]: goroutine 2339504 [IO wait, 2 minutes]:
Jul 14 05:13:00 venus caddy[833]: internal/poll.runtime_pollWait(0x7fe15e305b80, 0x72, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: runtime/netpoll.go:222 +0x55
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).wait(0xc0043c4998, 0x72, 0x200, 0x205, 0xffffffffffffffff)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:87 +0x45
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*pollDesc).waitRead(…)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_poll_runtime.go:92
Jul 14 05:13:00 venus caddy[833]: internal/poll.(*FD).Read(0xc0043c4980, 0xc000453b00, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: internal/poll/fd_unix.go:166 +0x1d5
Jul 14 05:13:00 venus caddy[833]: net.(*netFD).Read(0xc0043c4980, 0xc000453b00, 0x205, 0x205, 0x0, 0x3ed, 0xc0032296d8)
Jul 14 05:13:00 venus caddy[833]: net/fd_posix.go:55 +0x4f
Jul 14 05:13:00 venus caddy[833]: net.(*conn).Read(0xc001392110, 0xc000453b00, 0x205, 0x205, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: net/net.go:183 +0x91
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*atLeastReader).Read(0xc00137ddd0, 0xc000453b00, 0x205, 0x205, 0x80, 0xc00059c000, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:776 +0x63
Jul 14 05:13:00 venus caddy[833]: bytes.(*Buffer).ReadFrom(0xc002490278, 0x1d2d7c0, 0xc00137ddd0, 0x40b9c5, 0x1818a00, 0x198d260)
Jul 14 05:13:00 venus caddy[833]: bytes/buffer.go:204 +0xbe
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readFromUntil(0xc002490000, 0x1d31080, 0xc001392110, 0x5, 0xc001392110, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:798 +0xf3
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecordOrCCS(0xc002490000, 0x0, 0x0, 0x8000000000000000)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:605 +0x115
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).readRecord(…)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:573
Jul 14 05:13:00 venus caddy[833]: crypto/tls.(*Conn).Read(0xc002490000, 0xc00228e000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
Jul 14 05:13:00 venus caddy[833]: crypto/tls/conn.go:1276 +0x165
Jul 14 05:13:00 venus caddy[833]: net/http.(*connReader).Read(0xc00131dda0, 0xc00228e000, 0x1000, 0x1000, 0xc00077fb90, 0x64d971, 0xc001392110)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:800 +0x1b9
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).fill(0xc002af10e0)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:101 +0x108
Jul 14 05:13:00 venus caddy[833]: bufio.(*Reader).Peek(0xc002af10e0, 0x4, 0x45ff5bf68c45, 0x282b400, 0x0, 0x0, 0x282b400)
Jul 14 05:13:00 venus caddy[833]: bufio/bufio.go:139 +0x4f
Jul 14 05:13:00 venus caddy[833]: net/http.(*conn).serve(0xc002493400, 0x1d50178, 0xc001bf1880)
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:1977 +0xa47
Jul 14 05:13:00 venus caddy[833]: created by net/http.(*Server).Serve
Jul 14 05:13:00 venus caddy[833]: net/http/server.go:3013 +0x39b
Jul 14 05:13:00 venus systemd[1]: caddy.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Jul 14 05:13:00 venus systemd[1]: caddy.service: Failed with result ‘exit-code’.
5. What I already tried:
Nothing. Systemd automatically restarts Caddy and it’s fine.
Also reported on certmagic-s3 github