Out of memory issue with Caddy

Help
1

1. My Caddy version (caddy version):

We are using commit id : 2922d09bef3c504dde66bc12f7441668fcef6a20

2. How I run Caddy and the Problem

We develop a few plugins on top of Caddy to serve our purpose. This basically checks the request and shows some static contents. Recently we are seeing out of memory issue for our container and the container restarts after a few weeks because of that memory issue. It receives sever requests every few seconds. The memory limit for the container is 3.618GiB.

It is running as a docker container. We run it with docker run command with restart always.

Here is the error message:

^[[36mINFO^[[0m[2020/02/27 15:52:26] RequestURI:/wpad.dat
173.46.253.184 - - [27/Feb/2020:15:52:26 +0000] "GET /wpad.dat HTTP/1.1" 200 18403
^[[36mINFO^[[0m[2020/02/27 15:52:29] RequestURI:/bnb/operation
12.89.45.18 - - [27/Feb/2020:15:52:29 +0000] "POST /bnb/operation HTTP/1.1" 200 18403
fatal error: runtime: out of memory
 
runtime stack:
runtime.throw(0xbfd885, 0x16)
        /usr/local/go/src/runtime/panic.go:605 +0x95
runtime.sysMap(0xc4e9d30000, 0x100000, 0x41e100, 0x1092158)
        /usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0x1078a60, 0x100000, 0x7fbdd5aaf608)
        /usr/local/go/src/runtime/malloc.go:470 +0xd7
runtime.(*mheap).grow(0x1078a60, 0x4, 0x0)
        /usr/local/go/src/runtime/mheap.go:887 +0x60
runtime.(*mheap).allocSpanLocked(0x1078a60, 0x4, 0x1092180, 0xc4222b7db0)
        /usr/local/go/src/runtime/mheap.go:800 +0x334
runtime.(*mheap).allocManual(0x1078a60, 0x4, 0x1092180, 0xc4ccc13380)
        /usr/local/go/src/runtime/mheap.go:762 +0x53
runtime.stackpoolalloc(0x1090300, 0xc4e9d20000)

Any idea why I am facing this out of memory issue?

2

Interesting… is there any more to the output?

A lot of questions though, that need answering:

  • Are there any other processes running in the container?
  • Can you verify that Caddy is the one using all that memory?
  • What kinds of things (need to) happen to lead to this error?
  • What are your exact build steps?
  • What are the complete logs?
  • What is your complete and unaltered config?

This version is 2 years old… you should upgrade.

Since Caddy 1 has seen widespread production use without these errors, I’m going to go out on a limb and suggest this is the problem: a bug in a plugin. If not, maybe it’s a bug that was fixed long ago.

At this point, it just comes down to regular memory profiling and such.

You can enable the expvar and pprof directives to expose debug endpoints that will give you memory profiles and heap statistics.

Good luck!

3

Hi @matt,
Thanks for quick response. I found the root cause of the issue. It’s actually in our code which maintains some maps for requested domain and there is no size limit on that. This is causing that huge memory issue.

Thanks,
Mahfuz

3 Likes
4

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.