Blocking bad bots and bad browser requests

Help
1

I’ve tried hunting around and wasn’t able to find anything on this so maybe caddy handles this already and I don’t realize it.

On apache, I could set up bad bot lists using user-agent or ip and apache would immediately end the request without wasting server resources delivering pages to them.

Although I can handle some of this in php, that’s processing time and so on. I do have security software on my Wordpress sites, too. And it handles blocking bots, etc, too, but I see that as the last resort.

Does caddy have any built in security against bad bots, malicious requests (cloudflare calls them bad browsers)?

2

You can block IPs with the ipfilter plugin: https://caddyserver.com/docs/http.ipfilter

About blocking by user agent, you can rewrite their requests to something useless. See https://caddyserver.com/docs/rewrite and https://caddyserver.com/docs/placeholders.

Warning, I did not test this:

ipfilter / {
	rule block
	ip 70.1.128.0/19 2001:db8::/122 9.12.20.16
}
rewrite / {
	if     {>UserAgent} match Bot
	to     /something-that-doesnt-exist-and-will-generate-404
}
3

Thank you. That is a start.

Here is an example of what I’d like to use. https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/custom.d/globalblacklist.conf

Note, I wouldn’t want to waste even serving a 404 to a bad bot… would rather black hole the request or 403. Anything that kills wasting CPU cycles or bandwidth on the creep.

4

Hi @jbmanos,

Unfortunately there’s no way for you to not waste CPU cycles on this. If you do any filtering at all, that has to be computed somewhere, whether thats done by Caddy or some software you put in front of Caddy. Limiting your responses to 403s or similar is going to be your most efficient approach.

You’d have to do some significant parsing of that list to turn it into something Caddyfile-worthy. You’ll be making comparisons using Caddy’s {>Header} placeholder format. @lbguilherme has started you off with an excellent example.

Generally, where the file you linked takes the form:

BrowserMatchNoCase "^[STRING1]" bad_bot
BrowserMatchNoCase "^[STRING2]" bad_bot
...
SetEnvIfNoCase Referer ~*[STRING3] spam_ref=1
SetEnvIfNoCase Referer ~*[STRING4] spam_ref=1
...

You’d want the Caddyfile to take the form:

example.com

rewrite {
  // Change conditional operator so any individual match triggers the rewrite
  // The default behaviour is AND, so normally all the conditions would have to match!
  if_op or

  // List out all bad User-Agents:
  if {>User-Agent} starts_with [STRING1]
  if {>User-Agent} starts_with [STRING2]
  ...

  // List out all bad Referers:
  if {>Referer} ends_with [STRING3]
  if {>Referer} ends_with [STRING4]
  ...

  // Send them all to a hole:
  to /bad-robot
}

// And here's the hole to put them in:
status 403 /bad-robot

https://caddyserver.com/docs/placeholders
https://caddyserver.com/docs/rewrite
https://caddyserver.com/docs/status

5

thank you. this starts to beg for a way to include a file in the caddy file. I’m choking on regex today, but is there yet a way to do that?

6

Yep! You’re looking for import. Caddy will treat an import directive as though it were replaced by the contents of the file it indicates. It’s quite useful for shared website configuration, too.

rewrite {
  if_op or
  import useragents.conf
  import referers.conf
  to /bad-robot
}

Where the contents of useragents.conf, for example, would be:

  if {>User-Agent} starts_with [STRING1]
  if {>User-Agent} starts_with [STRING2]

https://caddyserver.com/docs/import

7

oh sweet! I have five wordpress sites on the machine and this will make adding all those file blocks and little security things much easier to update!!! nice!

8

This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.