CPU and memory usage are very high

[ablond (Alexandre)]

1. The problem I’m having:

CPU and memory usage are very high. I only have a few dozen requests per second,
and caddy consumes several GB of memory (1.5 on average, with peaks at 5GB). The CPU is constantly being used at 100% …

I’ve been using the souin cache with Badger since upgrading to 2.8.4 (due to a problem with redis).
But is that enough to explain such high cache consumption? Especially with so few requests, and a cache ttl of 300s?

Do you have an avenue to explore? I admit I have no ideas, and everything seems to be working fine. Except that performance-wise, it’s complicated to maintain.

I put 2 pprof links at the end of the post, one with the CPU profile, the other with the memory.
It seems to be “regexp.(*Regexp).MatchString” that consumes 80%+ of the CPU, but I don’t know more about it :/.

If you need more information, don’t hesitate

2. Error messages and/or full log output:

No particular error message, apart from “aborting with incomplete response”, “context cancelled”, …

3. Caddy version:

v2.8.4 h1:q3pe0wpBj1OcHFZ3n/1nl4V4bxBrYoSoab7rL9BMYNk=

4. How I installed and ran Caddy:

a. System environment:

The host machine runs Debian GNU/Linux 12

Caddy is launched in a docker container, via docker compose.

b. Command:

docker compose

services:
  caddy:
    build: docker/caddy
    restart: unless-stopped
    env_file:
      - caddy.env
    ports:
      - "80:80"
      - "2019:2019"
      - "443:443"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile
      - ./data:/data
      - ./certs:/certs
      - ./config:/config
    networks:
      - stg-proxy
  
networks:
  stg-proxy:
    enable_ipv6: true
    name: stg-proxy
    ipam:
      config:
        - subnet: fd00::/8

c. Service/unit/compose file:

FROM caddy:2.8.4-builder AS builder

RUN xcaddy build \
    --with github.com/caddy-dns/ovh \
    --with github.com/caddyserver/cache-handler

FROM caddy:2.8.4

COPY --from=builder /usr/bin/caddy /usr/bin/caddy

d. My complete Caddy config:

{
    order cache before rewrite

    cache {
        api {
            souin
        }
    }

    email alexandre@xxx.fr

    on_demand_tls {
        ask      http://php:80/api/domains/check
    }

    log {
        output stderr
        format json
        level ERROR
    }

    servers {
        metrics
    }
}

(generic_reverse_proxy) {
    @backendRoutes `{path}.startsWith("/api") || {path}.startsWith("/dashboard") || {path}.startsWith("/bundles") || {path}.startsWith("/medias")`

    encode {
        gzip
        zstd
    }

    handle @backendRoutes  {
        cache {
            ttl 300s
        }
        reverse_proxy php:80
    }

    reverse_proxy nextjs:3000
}

redirect-domain.fr {
    reverse_proxy php-mob-redirect:80
}

specific-domain.xxx.fr {
    tls /certs/custom-cert.crt /certs/custom-server-key.key
    import generic_reverse_proxy
}

main-domain.fr {
    redir https://www.{host}{uri}
}

*.main-domain.fr {
    tls {
        dns ovh {
            endpoint {$OVH_ENDPOINT}
            application_key {$OVH_APPLICATION_KEY}
            application_secret {$OVH_APPLICATION_SECRET}
            consumer_key {$OVH_CONSUMER_KEY}
        }
    }

    import generic_reverse_proxy
}

https:// {
    tls {
        on_demand
    }

    import generic_reverse_proxy
}

http://caddy {
    @denied not client_ip private_ranges
    abort @denied

    @metricspath `{path}.startsWith("/metrics")`

    metrics @metricspath

    reverse_proxy /debug/pprof/* localhost:2019 {
        header_up Host {upstream_hostport}
    }

    @souinapi `{path}.startsWith("/souin-api")`

    cache @souinapi {
    }
}

5. Links to relevant resources:

pprof profile : Pprof.me
(svg link)

pprof heap : Pprof.me
(svg link)

[matt (Matt Holt)]

/cc @darkweak