Caddy crashing with reverse proxy

1. The problem I’m having:

I am using caddy for reverse proxy on my node app, and recently it has been crashing (like the service) I’m not sure why. I get 80k users daily on my website and when it crashes the links I put in the Caddyfile go up after the restart but the Auto TLS/SSL links I set up take a long time to go up. (i have over 1k links pointing to my node app) I don’t know why this happens, can anyone help?

2. Error messages and/or full log output:

there isnt a specific error, it just tells me that the service is inactive and that it was killed when i do systemctl status caddy after it crashed. there are http reverse proxy errors, but i think that is normal, it doesnt seem to be doing anything (there are a ton of those errors)

3. Caddy version:

v2.7.6 h1:w0NymbG2m9PcvKWsrXO6EEkY9Ru4FJK8uQbYcev1p3A=

4. How I installed and ran Caddy:

i use the service to start it

sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf ‘https://dl.cloudsmith.io/public/caddy/stable/gpg.key’ | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf ‘https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt’ | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddy

a. System environment:

Debian Bullseye 11 x64

b. Command:

systemctl start caddy

c. Service/unit/compose file:

Caddyfile

d. My complete Caddy config:

{
        on_demand_tls {
                ask http://localhost:3000/check/
                interval 2m
                burst 5
        }
}

https://, :80 {
        tls {
                on_demand
        }
        reverse_proxy 127.0.0.1:8000
        @plausible path /js/script.js /api/event
        handle @plausible {
                rewrite /js/script.js /js/script.js
                reverse_proxy https://analytics.artclass.site {
                        header_up Host {http.reverse_proxy.upstream.hostport}
                }
        }
}


# HTTP domains
surfdoge.pro, dogesurf.app, derpman.lol {
reverse_proxy localhost:8000
}
# there are more links but i have too much to put

5. Links to relevant resources:

Maybe enable debug logs and see if that has anything ?

1 Like

Yeah - you’ll need to pull up your logs.

This line doesn’t do anything, you can remove it.

You can simplify this to header_up Host {upstream_hostport} which is a placeholder shortcut.

This doesn’t make sense – you shouldn’t have tls on a site on port :80. You should split this up.

I’m not sure I understand the intent though. Just let Caddy redirect HTTP->HTTPS automatically, no need to have :80 there.

Rate limiting is being deprecated, you should remove interval and burst here. It didn’t work properly anyway, and will cause you problems if you keep it.

We’ll need the logs (question 2 in the template) in order to help.

alright, i’ll fix the file. for the logs, i’m not quite sure if this is what your looking for, but here:

Apr 17 15:32:20 dogenetwork caddy[589353]: {"level":"error","ts":1713382340.3149827,"logger":"http.handlers.reverse_proxy","msg":"aborting with incomplete response","upstream":"localhost:8000","duration":0.230080819,"request":{"remote_ip":"204.185.144.215","remote_port":"26694","client_ip":"204.185.144.215","proto":"HTTP/2.0","method":"GET","host":"mathpractice.ugli.se","uri":"/bare/v1/","headers":{"X-Bare-Path":["/v3/signin/identifier?continue=https%3A%2F%2Fwww.youtube.com%2Ffavicon.ico&hl=en&ifkv=ARZ0qKJq8DW2eBm-2TcjYPxcy1S2kBBXu4viilM9AkD5y-S9cJLyDnOmj1dsOunY47BDYS6AIkx0&passive=true&service=youtube&uilel=3&flowName=GlifWebSignIn&flowEntry=ServiceLogin&dsh=S-871645523%3A1713382527010426&theme=mn&ddm=0"],"Userkey":["null"],"X-Bare-Host":["accounts.google.com"],"X-Bare-Forward-Headers":["[\"accept-encoding\",\"connection\",\"content-length\"]"],"X-Bare-Port":["443"],"Accept-Language":["en-US,en;q=0.9"],"Sec-Fetch-Mode":["same-origin"],"User-Agent":["Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0"],"Accept-Encoding":["gzip, deflate, br, zstd"],"X-Forwarded-Host":["mathpractice.ugli.se"],"Accept":["*/*"],"X-Bare-Protocol":["https:"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Dest":["empty"],"X-Bare-Headers":["{\"accept\":\"image/avif,image/webp,image/apng,image/svg+xml,image/*,*/*;q=0.8\",\"device-memory\":\"4\",\"dpr\":\"1.5\",\"sec-ch-ua\":\"\\\"Microsoft Edge\\\";v=\\\"123\\\", \\\"Not:A-Brand\\\";v=\\\"8\\\", \\\"Chromium\\\";v=\\\"123\\\"\",\"sec-ch-ua-full-version\":\"\\\"123.0.2420.97\\\"\",\"sec-ch-ua-full-version-list\":\"\\\"Microsoft Edge\\\";v=\\\"123.0.2420.97\\\", \\\"Not:A-Brand\\\";v=\\\"8.0.0.0\\\", \\\"Chromium\\\";v=\\\"123.0.6312.123\\\"\",\"sec-ch-ua-mobile\":\"?0\",\"sec-ch-ua-model\":\"\\\"\\\"\",\"sec-ch-ua-platform\":\"\\\"Windows\\\"\",\"sec-ch-ua-platform-version\":\"\\\"15.0.0\\\"\",\"user-agent\":\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/123.0.0.0 Safari/537.36 Edg/123.0.0.0\",\"viewport-width\":\"1272\",\"referer\":\"https://ww16.0123movie.net/search.html?q=batman\",\"cookie\":\"__Host-GAPS=1:aMLmfqHD39qthFZvxCpqGMaZrUiLbQ:ji2GKyVC4EcNe1Ak; 1P_JAR=2024-04-17-19; AEC=AQTF6HwIrwlLeXsO9P_0s10QCXTzLE1kLMIArEg7QoRYbKdBlWhm7LUvQQ; NID=513=OrfSz4CvFJkhiuuCGZbJrpuwCTX4tbGm_LJy5hNQ4pkGqon671j9XHQ2L7HCcFux6LOqtY6u-vwpQexsCGH8jgJLkAcNgJyr6xM3_2NeIyc8Iby4CD5A8JFw30eUwrRp5H7mn1_9e1xmphmcC3G0qwFNMfCfyjecCFmRMg3DEbXdPJRJeb2n_Yt75Qa8NEGfdwHWQEsLV7k; SNID=AOYECSpKKDzu1CnTl35g9Yxd9qHVz-1HYdYJOLEELWoqkGrUeYp74lsM24XWxp_G5lXcTqJpuwFMBZw6wgb41AQfssRjfz8rnMc\",\"Host\":\"accounts.google.com\"}"],"Referer":["https://mathpractice.ugli.se/sw.js"],"X-Forwarded-For":["204.185.144.215"],"X-Forwarded-Proto":["https"]},"tls":{"resumed":false,"version":772,"cipher_suite":4865,"proto":"h2","server_name":"mathpractice.ugli.se"}},"error":"reading: context canceled"}

Kind of, but a lot more lines :slight_smile: That particular line just means that either the server was closed or the connection got reset or the client did something, it doesn’t really mean much.

recently it has been crashing (like the service)

After it crashes, what is the recent log output shown by systemctl status caddy?

We need a lot more information than just vague “it crashed” and “it takes a long time” and “reverse proxy errors that are normal.” Please be specific.

my bad lol, im not really familiar with caddy. i will send the logs when it crashes again, i had restart on failure enabled so i wasn’t able to catch the log when it crashed.

edit: i found something else on the service, what does it mean?

     Loaded: loaded (/lib/systemd/system/caddy.service; disabled; vendor preset: enabled)
     Active: active (running) since Wed 2024-04-17 15:39:08 EDT; 2h 21min ago
       Docs: https://caddyserver.com/docs/
    Process: 597741 ExecReload=/usr/bin/caddy reload --config /etc/caddy/Caddyfile --force (code=exited, status=0/SUCCESS)>
   Main PID: 596051 (caddy)
      Tasks: 41 (limit: 14273)
     Memory: 1.0G
        CPU: 3h 37min 2.439s
     CGroup: /system.slice/caddy.service
             └─596051 /usr/bin/caddy run --environ --config /etc/caddy/Caddyfile

This says disabled here, which means the service will not start on boot. Did you disable the service at one point in time? One possibility of what happened is that your server rebooted (scheduled or on-demand) and Caddy wasn’t started on boot because its service is disabled.

For the logs, try journalctl -u caddy --no-pager | less +G and search for the time when you believe it crashed. Also run systemctl status caddy when the service is supposedly down. It’ll show information.

Okay, thanks, I will try that.
Also, is there a specific limit on how many domains caddy can reverse proxy and install certificates onto?

Not really, no. Just your system memory constraints, and ACME issuer rate limits, essentially. Tens to hundreds of thousands.

Alright, thanks.

also, I think this issue was fixed (kind of.) i found out that when you do caddy reload as soon as the service restarts all links will be up instantly, so when it crashes, it wont take forever to links to go back up. so it’s a half fix i guess.

thanks for the help!

I don’t understand what you mean by this.

Are you saying “caddy reload is graceful and doesn’t restart the server”? If so, yep, that’s the point of reload vs restart.

If not, can you clarify what you mean? What do you mean by “links”?