September 29, 2026: We have been seeing more sophisticated bot attacks bypassing our safeguards over the last few weeks. If the site is very slow or even appears to be down, bot attacks are most likely to blame. We are looking into ways to address the issue.
Subscribe to:
Post Comments (Atom)
12 comments:
It's down today. Any way of knowing from where? Not Yandex I suppose ;-)
The most recent generation of malicious bots uses US-based IP addresses, apparently procuring them from legitimate cloud services. It makes it much harder to protect against them :-(
What are they attacking?
We just happen to be one of thousands of medium-size Web sites with a lot of data that Web crawlers are hungry for. Al covered the gory details in an ISFDB Wiki post a few months ago -- see https://isfdb.org/wiki/index.php/ISFDB:Community_Portal#Google_linking_to_Stoecker.org_mirror_instead_of_ISFDB.org for details.
We need some vindictive fan of the site to create a hunter bot to go fry the servers the bots are coming from by causing all the fans to stop and preventing the systems from shutting down. This is getting ridiculous.
Unfortunately, it's not as simple as a single group of "servers the bots are coming from". If it were straightforward, Cloudflare, out front end, would be able to stop them easily. Instead it's a sophisticated operation using thousands of constantly changing IP addresses and AI agents pretending to be human. Cloudflare engineers continuously tweak anti-AI bot systems and we are currently looking into how we can best leverage them. Some options are very robust, but they are also expensive. Their "enterprise" tier of services starts in the "tens of thousands of dollars per year" range.
We can dream.
Alas, this does not load. I found stoecker.eu; password didn't work. I'll keep trying to load the article you referred to.
stoecker.eu is a partial mirror of the main ISFDB site, but it doesn't have the ISFDB Wiki mirrored. Here is the core of what Al's linked Wiki comment says: "Many folks may not be aware of the current ongoing invasion by AI web crawlers. These crawlers are using the same techniques as traditional DDOS attacks: Coordinated requests, originating from multiple sites/countries, involving hundreds of servers and IP addresses. Each individual server will request pages at a rate that bypasses rate limiting rules, and attempts to blacklist by IP address is not only manually intensive, but eventually those lists grow to such a length that the hosting system spends all of its time searching the blacklist instead of serving pages. These attacks do not honor the robot.txt file.
The attacks are having a profound impact on mid-tier websites that have modest hosting equipment, yet host large amounts of data. This impacts not only the ISFDB, but other sites with significant data, such as university libraries. [snip] Even companies and organizations which have hardened network firewalls are having difficulty dealing with the onslaught, and many sites have been placing restrictions on web scraping in general. The only way to keep the ISFDB operational is via the use of CloudFlare, where we have enabled its most agressive mode of defending against AI crawlers. That, however, is still not enough, and we have had to resort to blocking entire countries where the attacks originate. These countries shift as they are blocked, so I won't post which countries are currently on the blocked list. Cloudflare is currently experimenting with a beta program to charge the AI crawlers a per-page basis, or be denied access altogether. So there may be some relief in the future, but it is an ongoing arms race between web sites and web crawlers."
I’m so sorry that you’re being attacked like this. Can users help?
Unfortunately, there is nothing that ISFDB users can do to help, at least not at the moment. I will make sure to post an update if and when things change.
Such a shame! I was hoping to find out more about Edith Nesbit's horror collections :(
Post a Comment