Runtime: failed to create new OS thread [FIXED]

Caddy used to work and now it won’t start no more?? Any idea. The load used to be very high and now it’s low…

Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime: failed to create new OS thread (have 5 already; errno=11)
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime: may need to increase max user processes (ulimit -u)
Sep 28 14:42:43 srvz-webapp caddy[23684]: fatal error: newosproc
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime: failed to create new OS thread (have 6 already; errno=11)
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime: may need to increase max user processes (ulimit -u)
Sep 28 14:42:43 srvz-webapp caddy[23684]: fatal error: newosproc
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime stack:
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.throw(0xa29de1, 0x9)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/panic.go:566 +0x95
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.newosproc(0xc4200ee000, 0xc4200fffc0)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/os_linux.go:160 +0x194
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.newm(0xa7e7c8, 0xc420015500)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1572 +0x132
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.startm(0xc420015500, 0x100000001)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1642 +0x181
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.handoffp(0xc420015500)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1680 +0x296
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.entersyscallblock_handoff()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:2453 +0x3a
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.systemstack(0xc420014000)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/asm_amd64.s:298 +0x79
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.mstart()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1079
Sep 28 14:42:43 srvz-webapp caddy[23684]: goroutine 17 [syscall]:
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.notetsleepg(0xd1b5f8, 0x34630b78225, 0x0)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/lock_futex.go:205 +0x42 fp=0xc420018740 sp=0xc420018710
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.timerproc()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/time.go:209 +0x2eb fp=0xc4200187c0 sp=0xc420018740
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.goexit()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/asm_amd64.s:2086 +0x1 fp=0xc4200187c8 sp=0xc4200187c0
Sep 28 14:42:43 srvz-webapp caddy[23684]: created by runtime.addtimerLocked
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/time.go:116 +0xee
Sep 28 14:42:43 srvz-webapp caddy[23684]: goroutine 1 [runnable, locked to thread]:
Sep 28 14:42:43 srvz-webapp caddy[23684]: golang.org/x/net/http2/hpack.addDecoderNode(0x3ffffe0000005c0, 0xc4200ce002)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /home/caddy/go/src/golang.org/x/net/http2/hpack/huffman.go:137 +0x209
Sep 28 14:42:43 srvz-webapp caddy[23684]: golang.org/x/net/http2/hpack.init.1()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /home/caddy/go/src/golang.org/x/net/http2/hpack/huffman.go:127 +0x8a
Sep 28 14:42:43 srvz-webapp caddy[23684]: golang.org/x/net/http2/hpack.init()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /home/caddy/go/src/golang.org/x/net/http2/hpack/tables.go:353 +0x25a2
Sep 28 14:42:43 srvz-webapp caddy[23684]: github.com/lucas-clemente/quic-go/h2quic.init()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /home/caddy/go/src/github.com/lucas-clemente/quic-go/h2quic/server.go:230 +0x38
Sep 28 14:42:43 srvz-webapp caddy[23684]: github.com/mholt/caddy/caddyhttp/httpserver.init()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /tmp/custombuild_9236_003557283/src/github.com/mholt/caddy/caddyhttp/httpserver/vhosttrie.go:161 +0xd0
Sep 28 14:42:43 srvz-webapp caddy[23684]: github.com/mholt/caddy/caddyhttp.init()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /tmp/custombuild_9236_003557283/src/github.com/mholt/caddy/caddyhttp/caddyhttp.go:31 +0x32
Sep 28 14:42:43 srvz-webapp caddy[23684]: github.com/mholt/caddy/caddy/caddymain.init()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /tmp/custombuild_9236_003557283/src/github.com/mholt/caddy/caddy/caddymain/run.go:264 +0x6a
Sep 28 14:42:43 srvz-webapp caddy[23684]: main.init()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /tmp/custombuild_9236_003557283/src/github.com/mholt/caddy/caddy/main.go:15 +0x2e
Sep 28 14:42:43 srvz-webapp caddy[23684]: goroutine 5 [syscall]:
Sep 28 14:42:43 srvz-webapp caddy[23684]: os/signal.signal_recv(0x0)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/sigqueue.go:116 +0x157
Sep 28 14:42:43 srvz-webapp caddy[23684]: os/signal.loop()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/os/signal/signal_unix.go:22 +0x22
Sep 28 14:42:43 srvz-webapp caddy[23684]: created by os/signal.init.1
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/os/signal/signal_unix.go:28 +0x41
Sep 28 14:42:43 srvz-webapp caddy[23684]: goroutine 6 [select]:
Sep 28 14:42:43 srvz-webapp caddy[23684]: github.com/mholt/caddy/caddytls.maintainAssets(0xc42000a780)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /tmp/custombuild_9236_003557283/src/github.com/mholt/caddy/caddytls/maintain.go:47 +0x498
Sep 28 14:42:43 srvz-webapp caddy[23684]: created by github.com/mholt/caddy/caddytls.init.2
Sep 28 14:42:43 srvz-webapp caddy[23684]: /tmp/custombuild_9236_003557283/src/github.com/mholt/caddy/caddytls/maintain.go:18 +0x58
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime stack:
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.throw(0xa29de1, 0x9)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/panic.go:566 +0x95
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.newosproc(0xc420023000, 0xc420033fc0)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/os_linux.go:160 +0x194
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.newm(0x0, 0xc420014000)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1572 +0x132
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.startm(0xc420014000, 0x7fa5db2eb000)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1642 +0x181
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.handoffp(0xc420014000)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1669 +0x55
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.stoplockedm()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:1737 +0x160
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.schedule()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:2078 +0x2fa
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.goschedImpl(0xc4200001a0)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:2198 +0x100
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.gopreempt_m(0xc4200001a0)
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/proc.go:2213 +0x36
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.newstack()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/stack.go:1053 +0x318
Sep 28 14:42:43 srvz-webapp caddy[23684]: runtime.morestack()
Sep 28 14:42:43 srvz-webapp caddy[23684]: /usr/local/go/src/runtime/asm_amd64.s:366 +0x7f
Sep 28 14:42:43 srvz-webapp systemd[1]: caddy.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 28 14:42:43 srvz-webapp systemd[1]: Unit caddy.service entered failed state.
Sep 28 14:42:43 srvz-webapp systemd[1]: Dependency failed for Wait for Network to be Configured.

Looks like the system isn’t allowing Caddy to create enough threads. What’s your ulimit -u?

ulimit -u
7975

and

ps -eLf | wc -l
850

Indeed, too much stuff running on it…

Hmm happened for me too this night.

# ps -eLf | wc -l
531
# ulimit -u
48044

```