Server got hacked

1. The problem I’m having:

My caddy websites got hacked. The index.php file on all of my sites was replaced. I also found a Zero.php file on many but not all of my websites.

Is there any way to figure out what the root cause of this was?

2. Error messages and/or full log output:

No error messages

3. Caddy version:

v2.6.2 h1:wKoFIxpmOJLGl3QXoo6PNbYvGW4xLEgo32GPBEjWL8o=

4. How I installed and ran Caddy:

Installed via apt get. I am running with a couple of plugins:

GOOS=linux GOARCH=amd64 xcaddy build --with GitHub - greenpau/caddy-git: Git Plugin for Caddy v2 --with GitHub - caddy-dns/cloudflare: Caddy module: dns.providers.cloudflare

a. System environment:

Ubuntu

b. Command:

service caddy restart

d. My complete Caddy config:

Many websites different configs. Here is the main config.

#----------------
# Snippets
#----------------
(logroll) {
	roll_size 3MiB
	roll_keep 5
	roll_keep_for 48h
}
(errors-rewrite) {
	handle_errors {
		@404 `{err.status_code} == 404`
		rewrite @404 /404.html
		file_server
	}
}
(errors-basic) {
	handle_errors {
		respond {http.error.status_code} {http.error.status_text}
		header Content-Type "text/plain; charset=utf-8"
	}
}
(webhook) {
	@webhook {
		method POST
		path /update
	}
	respond @webhook "OK" 200 {
		close
	}
}
(php82) {
	php_fastcgi unix//run/php/php8.2-fpm.sock {
		try_files {path} {path}/index.php {path}/index.html =404
	}
	encode gzip zstd
	file_server
	push
}
(php81) {
	php_fastcgi unix//run/php/php8.1-fpm.sock {
		try_files {path} {path}/index.php {path}/index.html =404
	}
	encode gzip zstd
	file_server
	push
}
(www-redirect) {
	redir https://www.{host}{uri}
}
(pwa) {
	route /pwa.min.js {
		header {
			-Expires
			Cache-Control max-age=0
		}
	}
	rewrite /manifest.json /manifest/index.php
}
(default-headers) {
	header {
		-X-Powered-By
		-ETag
		?Cache-Control max-age=3600
		?Permissions-Policy interest-cohort=()
		?X-Frame-Options SAMEORIGIN
		?X-Xss-Protection 1;mode=block
		?X-Content-Type-Options nosniff
		?Referrer-Policy no-referrer-when-downgrade
		?Feature-Policy "camera 'none'; microphone 'none'"
		?X-UA-Compatible IE=edge
		?Content-Security-Policy "default-src * blob:; connect-src * performance.typekit.net; frame-src https://*; img-src https://* http://* * p.typekit.net data:; object-src 'none'; script-src 'self' https://* http://* * 'unsafe-inline' 'unsafe-eval' use.typekit.net; style-src * https://* 'unsafe-inline' use.typekit.net; upgrade-insecure-requests; font-src 'self' fonts.gstatic.com assets.swarmcdn.com use.typekit.net *.joeworkman.net *.weavers.space *.aspecthq.com data:"
	}
}
(block-wordpress) {
	@wordpress {
		path_regexp wp-\w+
	}
	respond @wordpress "No Wordpress here. Piss Off!!!" 404
}
(dotfiles) {
	@dotfiles {
		not path /.well-known/*
		path_regexp (^|\/)\.[^\.\/]+
	}
	respond @dotfiles 404
}
(site-setup) {
	header {
		-X-Powered-By
		-ETag
		Cache-Control max-age=3600
		Permissions-Policy interest-cohort=()
		X-Frame-Options SAMEORIGIN
		# X-Xss-Protection 1;mode=block
		X-Content-Type-Options nosniff
		Referrer-Policy no-referrer-when-downgrade
		Feature-Policy "camera 'none'; microphone 'none'"
		X-UA-Compatible IE=edge
		Content-Security-Policy "default-src * blob:; connect-src * performance.typekit.net; frame-src https://*; img-src https://* http://* * p.typekit.net data:; object-src 'none'; script-src 'self' https://* http://* * 'unsafe-inline' 'unsafe-eval' use.typekit.net; style-src * https://* 'unsafe-inline' use.typekit.net; upgrade-insecure-requests; font-src 'self' fonts.gstatic.com assets.swarmcdn.com use.typekit.net *.joeworkman.net *.weavers.space *.aspecthq.com data:"
	}

	@webhook {
		method POST
		path /update
	}
	respond @webhook "OK" 200 {
		close
	}

	route /pwa.min.js {
		header {
			-Expires
			Cache-Control max-age=0
		}
	}
	rewrite /manifest.json /manifest/index.php

	@wordpress {
		path_regexp wp-\w+
	}
	respond @wordpress "No Wordpress here. Piss Off!!!" 404

	@dotfiles {
		not path /.well-known/*
		path_regexp (^|\/)\.[^\.\/]+
	}
	respond @dotfiles "File Not Found" 404
}

#----------------
# Global Config
#----------------
{
	#debug
	email {env.EMAIL}
	acme_dns cloudflare {env.CLOUDFLARE_CHALLENGE}
	order git before respond
	git {
		import /etc/caddy/*.caddy.git
	}
}

#----------------
# Import Sites
#----------------
import /etc/caddy/*.caddy

5. Links to relevant resources:

One site that I will leave in this state for troubleshooting.

Since I use git to deploy my sites, it was easy to identify what files were new/modified. I have found the offending connections in the access logs. However, I am wondering how the world he got these php files on to my server.

I need to find the root case so that this does not happen again. Are there any known Caddy exploits with the version that I am running?

Interesting. The attacker is basically running uname -a. You are using an old version of Caddy, but I’m not aware of any particular vulnerabilities in our code. And since Caddy is written in Go, it’s virtually impossible for attackers to remotely exploit memory vulnerabilities that are one common root of defacements.

I’m going to guess it is an issue with a PHP app or possibly a bug in a third-party plugin.

I can see that your hotsname, nycaddy (?) is the same as in this issue where you commented:

And I can see from the config that you are using the git plugin. I am not very familiar with the plugin so it’s possible the plugin is configured insecurely? Or it is a bug in the plugin?

We will need much more information in order to help assess.

What is your full config?

What are your full and complete logs?

What files were changed?

What app(s) are you running in the backends? I see lots of PHP there, “no wordpress” – is there really no WordPress? WordPress and its plugin ecosystem are a very common attack vector, especially since we know the software on the server hasn’t been kept up-to-date.

Is https://www.weavers.space your site too?

Thank @matt. I have finally found the loop hole. It was my own fault in trusting mankind.

I have a product called Total CMS (hence you will never see me use Wordpress). While in real life, the admin area is on lockdown requiring authentication, the online demo is open for people to edit the site and play with the CMS.

In that demo, I have a file upload utility. A PHP file was uploaded and the trouble spidered out from there.

Yes. Weaver’s Space is my site. :smile:

If anyone is running PHP, make sure that this is in your php.ini file.

disable_functions = exec,system
2 Likes

Excellent, thanks for the detective work and the follow-up!

This is not a perfect solution, it’s barely a plug.

Best to make sure that upload directories cannot execute PHP at all, and are only served by file_server.

1 Like

100%. I am working on an update to my CMS to ensure this does not happen again.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.