Newbie getting random 502's

[pooriar]

1. Caddy version:

v2.6.2

2. How I installed, and run Caddy:

sudo apt install caddy

systemctl start caddy

a. System environment:

debian 11 Linode VPS, no docker, using systemd

b. Command:

systemctl start caddy

c. Service/unit/compose file:

(here’s the service file that spins up the rails app)

[Unit]
Description=Tools
After=network.target

[Service]
Type=simple
WatchdogSec=10
WorkingDirectory=/var/www/tools/current
ExecStart=/root/.rbenv/shims/bundle exec rails server -e production --port 3002
Restart=always

[Install]
WantedBy=multi-user.target

d. My complete Caddy config:

# The Caddyfile is an easy way to configure your Caddy web server.
#
# Unless the file starts with a global options block, the first
# uncommented line is always the address of your site.
#
# To use your own domain name (with automatic HTTPS), first make
# sure your domain's A/AAAA DNS records are properly pointed to
# this machine's public IP, then replace ":80" below with your
# domain name.

www.pooriar.com, pooriar.com {
	# Set this path to your site's directory.
	root * /var/www/pooriar/current/_site

	# Enable the static file server.
	file_server
	try_files {path}.html

	# Another common task is to set up a reverse proxy:
	# reverse_proxy localhost:8080

	# Or serve a PHP site through php-fpm:
	# php_fastcgi localhost:9000
}

wedding.pooriar.com/assets/* {
	root * /var/www/wedding-ai/current/public/
  file_server
}

wedding.pooriar.com {
  reverse_proxy localhost:3001
}

tools.pooriar.com {
  reverse_proxy localhost:3002
}

tools.pooriar.com/assets/* {
	root * /var/www/tools/current/public/
  file_server
  log {
    output file /var/log/caddy/tools-access.log
  }
}
# Refer to the Caddy docs for more information:
# https://caddyserver.com/docs/caddyfile

3. The problem I’m having:

I’ve got a static site at pooriar.com and a rails site at tools.pooriar.com

Anytime I refresh tools.pooriar.com, there’s a small but significant chance of getting a 502 error, which lingers for a few seconds. If I wait a few seconds and keep refreshing, things go back to normal.

If I tail the rails logs while doing this, I can see that the rails app doesn’t go down, it doesn’t even receive any request at all during those 502’s

4. Error messages and/or full log output:

Jan 23 16:21:39 localhost caddy[713511]: {"level":"error","ts":1674490899.91727,"logger":"http.log.error.log0","msg":"dial tcp [::1]:3002: connect: connection refused","request":{"remote_ip":"2806:2f0:8100:274:d65:f7ab:7eab:3e4","remote_port":"55533","proto":"HTTP/3.0","method":"GET","host":"tools.pooriar.com","uri":"/","headers":{"Sec-Ch-Ua":["\"Not?A_Brand\";v=\"8\", \"Chromium\";v=\"108\", \"Google Chrome\";v=\"108\""],"Accept":["text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"],"Accept-Language":["en-US,en;q=0.9"],"User-Agent":["Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/108.0.0.0 Safari/537.36"],"Sec-Fetch-Site":["same-origin"],"Sec-Fetch-Mode":["navigate"],"Cookie":[],"Sec-Ch-Ua-Mobile":["?0"],"Sec-Fetch-Dest":["document"],"Accept-Encoding":["gzip, deflate, br"],"Cache-Control":["max-age=0"],"Sec-Ch-Ua-Platform":["\"macOS\""],"Upgrade-Insecure-Requests":["1"],"Sec-Fetch-User":["?1"]},"tls":{"resumed":false,"version":0,"cipher_suite":0,"proto":"","server_name":""}},"duration":0.00096623,"status":502,"err_id":"gc6mnet1u","err_trace":"reverseproxy.statusError (reverseproxy.go:1272)"}

5. What I already tried:

Restarted Caddy
Tailed the Rails application logs - no issues there

6. Links to relevant resources:

[pooriar]

I’m totally new to all this stuff so thank you so much to anyone who can help

[RiversideRocks]

@pooriar

tcp [::1]:3002: connect: connection refused

It looks like your backend/service running on port 3002 is not online. Can you take a look at that?

[pooriar]

So port 3002 is where my rails server is running, and I was watching my rails app logs to see if there were any errors or anything, but there weren’t.

When that 502 happens, rails doesn’t log the requests at all, and I don’t think it’s going down either, since it wouldn’t be back up so fast, and there’s no error or anything that would take it down. It’s as if the request isn’t even making it to rails in those cases.

[pooriar]

Figured out the problem. It wasn’t an issue with Caddy, nor with Rails. The issue was that I had this WatchdogSec=10 in my systemd service, which I didn’t actually understand.
That expects a notification every 10 seconds, and when it doesn’t get it, it restarts the service. I wasn’t sending the expected notification. So my service was just restarting the rails app constantly, which explains the intermittent downtime.

Removing the watchdogsec line fixed it.

[francislavoie]

@pooriar

wedding.pooriar.com/assets/* {
	root * /var/www/wedding-ai/current/public/
  file_server
}

wedding.pooriar.com {
  reverse_proxy localhost:3001
}

FYI, using a path matcher in the site address is deprecated, and will be removed at some point in the future.

Instead, you should use handle blocks like this:

wedding.pooriar.com {
	handle /assets/* {
		root * /var/www/wedding-ai/current/public
		file_server
	}

	handle {
		reverse_proxy localhost:3001
	}
}