Strange epmd caddy zombie process

1. The problem I’m having:

There is a strange epmd caddy zombie process and I have no idea how to find out what is causing it. Nor how to debug it and it is driving me crazy. Can anyone help?

2. Error messages and/or full log output:

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
caddy       6033  0.0  0.0      0     0 ?        Zs   12:00   0:00 [epmd] <defunct>

3. Caddy version:

v2.8.4

4. How I installed and ran Caddy:

I downloaded caddy binary with cloudflare and crowdsec modules from here Download Caddy, then I followed step by step install guide from static binaries here Install — Caddy Documentation including creating of caddy user and systemd scripts.

a. System environment:

Hetzner CAX31 Ampere aarch64 Cloud VPS with Ubuntu 24.04 LTS

b. Command:

It is starting through systemd so normally i use only this for regular maintanance

sudo vi /etc/caddy/Caddyfile
sudo caddy fmt -w /etc/caddy/Caddyfile
sudo caddy validate --config /etc/caddy/Caddyfile
sudo systemctl stop/start/reload/restart caddy.service

d. My complete Caddy config:

##########GLOBAL OPTIONS##########
{
	acme_ca https://acme-v02.api.letsencrypt.org/directory
	acme_dns cloudflare ****************************************
	email name.surname@example.com
	http_port 80
	https_port 443
	crowdsec {
		api_url http://localhost:8080
		api_key ****************************************
		ticker_interval 15s
	}
	log default {
		output stdout
		format json
		exclude http.log.access
		output file /var/log/caddy/access.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
}

##########PHP APPS##########

www.example1.com {
	redir * https://example1.com{uri}
}

www.example2.com {
	redir * https://example2.com{uri}
}

example1.com {
	file_server
	root * /srv/www/clients/example1.com/html/
	header {
		Strict-Transport-Security max-age=63072000
		X-Frame-Options "SAMEORIGIN"
		X-Content-Type-Options nosniff
		X-XSS-Protection "1; mode=block"
		Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()"
		Referrer-Policy "strict-origin"
		Expect-CT "max-age=604800"
	}
	log {
		output file /srv/www/clients/example1.com/logs/caddy_access.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
	php_fastcgi unix//var/run/php/php8.3-fpm.sock
	encode {
		zstd
		gzip
	}
}

example2.com {
	file_server
	root * /srv/www/clients/example2.com/html/
	header {
		Strict-Transport-Security max-age=63072000
		X-Frame-Options "SAMEORIGIN"
		X-Content-Type-Options nosniff
		X-XSS-Protection "1; mode=block"
		Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()"
		Referrer-Policy "strict-origin"
		Expect-CT "max-age=604800"
	}
	log {
		output file /srv/www/clients/example2.com/logs/caddy_access.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
	php_fastcgi unix//var/run/php/php8.3-fpm.sock
	encode {
		zstd
		gzip
	}
}

db.example.com {
	redir * https://db.example.com:8443{uri}
}

https://db.example.com:8443 {
	log {
		output file /srv/www/admin/db.example.com/logs/caddy_access.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
	file_server
	root * /srv/www/admin/db.example.com/html
	header {
		Strict-Transport-Security max-age=63072000
		Expect-CT "max-age=604800"
		Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()"
		X-Robots-Tag "noindex,nofollow"
	}
	php_fastcgi unix//var/run/php/php8.3-fpm.sock
	encode {
		zstd
		gzip
	}
}

https://nextcloud.example.com:8443 {
	log {
		output file /srv/www/admin/nextcloud.example.com/logs/caddy_access.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
	encode zstd gzip
	root * /srv/www/admin/nextcloud.example.com/html/

	redir /.well-known/carddav /remote.php/dav/ 301
	redir /.well-known/caldav /remote.php/dav/ 301
	redir /.well-known/* /index.php{uri} 301 # Nextcloud front-controller handles routes to /.well-known
	redir /remote/* /remote.php{uri} 301
	redir /.well-known/webfinger /index.php/.well-known/webfinger 301
	redir /.well-known/nodeinfo /index.php/.well-known/nodeinfo 301

	header {
		Strict-Transport-Security max-age=31536000
		Permissions-Policy interest-cohort=()
		X-Content-Type-Options nosniff
		X-Frame-Options SAMEORIGIN
		Referrer-Policy no-referrer
		X-Robots-Tag: "noindex,nofollow"
		X-XSS-Protection "1; mode=block"
		X-Permitted-Cross-Domain-Policies none
		X-Powered-By
	}

	php_fastcgi unix//var/run/php/php8.3-fpm.sock {
		env front_controller_active true
		env modHeadersAvailable true
	}

	@forbidden {
		path /build/* /tests/* /config/* /lib/* /3rdparty/* /templates/* /data/*
		path /.* /autotest* /occ* /issue* /indie* /db_* /console*
		not path /.well-known/*
	}

	error @forbidden 404

	@immutable {
		path *.css *.js *.mjs *.svg *.gif *.png *.jpg *.ico *.wasm *.tflite
		query v=*
	}

	header @immutable Cache-Control "max-age=15778463, immutable"

	@static {
		path *.css *.js *.mjs *.svg *.gif *.png *.jpg *.ico *.wasm *.tflite
		not query v=*
	}

	header @static Cache-Control "max-age=15778463"

	@woff2 path *.woff2
	header @woff2 Cache-Control "max-age=604800"

	file_server
}

##########DOCKER###########

portainer.example.com {
	redir * https://portainer.example.com:8443{uri}
}

https://portainer.example.com:8443 {
	log {
		output file /srv/www/admin/portainer.example.com/logs/caddy_access.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
	reverse_proxy localhost:9000
	header {
		Strict-Transport-Security max-age=63072000
		X-Frame-Options "SAMEORIGIN"
		X-Content-Type-Options nosniff
		X-XSS-Protection "1; mode=block"
		Expect-CT "max-age=604800"
		Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()"
		X-Robots-Tag "noindex,nofollow"
	}
}

analytics.example.com {
	log {
		output file /srv/www/admin/analytics.example.com/logs/caddy_access.log {
			roll_size 1gb
			roll_keep 5
			roll_keep_for 720h
		}
	}
	reverse_proxy localhost:8000
	header {
		Strict-Transport-Security max-age=63072000
		Referrer-Policy "strict-origin"
		Expect-CT "max-age=604800"
		Permissions-Policy "accelerometer=(), ambient-light-sensor=(), autoplay=(), battery=(), camera=(), cross-origin-isolated=(), display-capture=(), document-domain=(), encrypted-media=(), execution-while-not-rendered=(), execution-while-out-of-viewport=(), fullscreen=(), geolocation=(), gyroscope=(), keyboard-map=(), magnetometer=(), microphone=(), midi=(), navigation-override=(), payment=(), picture-in-picture=(), publickey-credentials-get=(), screen-wake-lock=(), sync-xhr=(), usb=(), web-share=(), xr-spatial-tracking=(), clipboard-read=(), clipboard-write=(), gamepad=(), speaker-selection=(), conversion-measurement=(), focus-without-user-activation=(), hid=(), idle-detection=(), interest-cohort=(), serial=(), sync-script=(), trust-token-redemption=(), window-placement=(), vertical-scroll=()"
		X-Robots-Tag "noindex,nofollow"
	}
}

5. Links to relevant resources:

N/A

Hi, have been able to find what is causing it. In the previous post a shared the config where analytics.example.com host is reverse proxy for Plausible Analytics running in docker. When I stop plausible, zombie process disapears, when I start it, it appears again. The PID 28437 is a parent of 285542. I am not sure whether it is a caddy or rather plausible issue. Whether should I care or ignore it?

USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
caddy     284307 30.4  2.7 1563972 440200 ?      Ssl  10:01   0:10 /app/erts-14.2.1/bin/beam.smp -- -root /app -bindir /app/erts-14.2.1/bin -progname erl -- -home /home/plausible -- -noshell -s elixir start_cli -mode embedded -setcookie TCEUJAMHIJBNHOLQMESR4MCUZNUUZBFQ6AKSALWRD5RU5SNWNFSA==== -sname plausible -config /app/releases/0.0.1/sys -boot /app/releases/0.0.1/start -boot_var RELEASE_LIB /app/lib -- -extra --no-halt
caddy     285542  0.0  0.0      0     0 ?        Zs   10:02   0:00 [epmd] <defunct>

To be clear, you’re looking at the USER column, which is not the program running but rather the username of the user under which that process is running.

I think plausible is using the caddy user for some reason, possibly because the user ID it uses overlaps with the one the Caddy package created? I’m not sure how plausible works.

1 Like

Hi @francislavoie many thanks for your response. There are only two processes running as caddy user on my VPS. It is caddy and php-fpm. Php-fpm is not the case. Caddy is acting as reverse proxy for Plausible Analytics web and measuring JS. Plausible Analytics are running in docker container. The response I have from the developer of Plausible Analytics is a sense that it is kind of usual behaviour of EPMD, but I have not been able to find any references to this.

Docker containers can run as any user. It might be that it picked the same user ID as Caddy’s user. Run id caddy and you’ll what Caddy’s user is. For me it’s 999 because it’s the first non-regular-user which was created on my system (i.e. user ID 1000). Plausible is likely creating a user inside the container and they happened to choose ID 999 as well. So yeah, just normal misleading Linux stuff.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.