Context:
I added on demand tls and due to the nature of my site (anyone can use any domain/subdomain they want, it’s a requirement, not great, don’t ask).
We also have wildcard DNS on our domain.
Recently our site was having some issues with speed and I checked our caddy logs, and I saw a bunch of on demand requests for random things like admin.hdbah.mydomaij.tld, all of them had something to do with an admin dashboard for different services. The slowness was due to another issue we had, and it went away when we resolved it.
The spam doesn’t seem to slow our servers at all because of their high capacity, but I find it really weird with this spam mainly because of the IP’s it’s coming from. Our site is relatively small but the attack seems to be coming from a bot net, I’ve blocked over 30 IP’s and the rate hasn’t dropped, so the attacker is either using rotating proxies or just a ton of servers. This isn’t effecting us from what I’ve seen, but I’m really curious if anyone else has seen something similar, I’m wondering if it could be a crawler that finds sites setup like mine and attempts to get to the admin pages? Not sure, we don’t have an admin page to begin with so we are fine.
Anyone have ideas?
You’re right, that’s not great, and I will have to ask. Because you’re literally turning off the mechanism of protection that we warn about being necessary in the docs.
You MUST reject unrecognized domain names in your ask endpoint. If you don’t, then you’re open to DDOS like you’re experiencing now.
You MUST keep track of a list of domains you want to allow, by having users pre-register their domains in some authenticated way with your service before adding it to the list.
Sounds like you’re experiencing a (relatively mild) DDoS attack. There are services that detect and block these, such as Cloudflare for example. Unfortunately this is a normal part of running a website these days.
FWIW, our on-demand TLS demo on the Caddy homepage does not hard-code a list of authorized domains, per-se, but we do limit it to a specific subdomain of a registered domain. This makes the costs of such attacks quite high, as you can get only one connection per (presumably paid-for) domain name.
Other than using something like Cloudflare (which you’d have to pass the TCP packets through for on-demand TLS), or enabling Caddy’s built-in protections, I don’t know what to tell you. If you disable the anti-abuse features your server will inevitably be abused.
I am logging all the domains to a database using the ask endpoint, I would be more worried if this was slowing down our servers, but it is easier for the users this way and isn’t effecting anything so far.
My main question was if anyone else has experienced this, because it seems like a full on botnet from logs and has been adapting very fast, for example I disabled very long domains, only allowing a maximum of 4 subdomains strung together, for example x.x.x.x would work but y.y.y.y.y wouldn’t (some of our users use freedns and those domains can get long, that’s why I chose 4) as the bots seemed to spam up to 7 or 8. Within 5 minutes the bots had started spamming shorter urls that fit within this.
The community I’m in that uses my site is relatively small and made up mostly of younger kids, so I don’t know anyone who would have the capabilities to do an attack with a bot net. Mainly posted here because it seems interesting
Update after typing the original part: I was talking with my dad about this, and at his work firewall logs are also spammed with a similar thing, seems like my idea was right, some bot going around spamming random domains looking for any admin panels
We would use cloudflare, but our service requires anyone to be able to make a domain (hence the autossl). It’s for a site that is used to bypass censorship in other countries, and they mainly block off domain, so we want anyone to just point it to our IP and have it work instantly. Currently the attack isn’t effecting our speeds from what I can see, so I’m just going to ignore it, but if you have any ideas on ddos protection I would be down to hear them
What you’re experiencing is very much expected, and that’s why you MUST NOT allow all domains, and ONLY allow specifically-registered domains.
You will need to change how your system operates to reject all domains by default and only allow specific domains that your users manually register with your system. This is how everyone else who uses On-Demand TLS uses it, and it’s necessary to avoid DDoS.
If you continue to allow this, you’re causing stress to the ACME issuers by issuing completely useless certificates, which will also sit in your server’s storage for 3 months taking up storage space. You also risk hitting ACME rate limits and getting your IP address blocked from issuing more certs.
If your server is connected to the Internet, you don’t need to know anyone. That’s just how it goes when you open a server to the Internet. The entire IPv4 space of the Internet can be scanned in a matter of seconds/minutes, and there are random bots out there just trying exploits or attacks constantly.
Even with a single subdomain of a registered domain (i.e. x.y.z where x is the subdomain, and x.y is the registered domain (with y being the TLD)) allows for nearly infinite combinations of domains to attack with.
Like I mentioned above, if you don’t want to explicitly allow every domain to issue for, you need to do something like we do on our website, which is restricting issuance to a single subdomain (with a specific value) of registered domains only; that means no arbitrary subdomains, and no sub-subdomains, etc.
I’m not really worried about storage or the DDoS, as neither of them affect the server (we have tons of storage, barely use any of it to start with, and our servers performance doesn’t seem to change at all after the spam started). We also want the most seamless experience, along with the potential of the domain that has the registration being censored (The whole point is to uncensor things).
Although Matt Holt’s idea seems like it could work, only allowing one ssl per domain, I’ll modify it a little bit due to some users using freedns and using duplicate domains, but the same idea.
