Get the latest tech news

Abusive AI Web Crawlers: Get Off My Lawn

many other folks have reported in the last few weeks, we have also been seeing a huge increase in the amount of traffic from abusive web crawlers. Automated blocking of abusive traffic has long been a necessary evil.

These are typically attempting to brute force passwords or run web security scanners, and we firewall IP addresses out after a number of suspicious requests. These crawlers and bad-actors can already outnumber real organic traffic but the scale of the recent activity, along with the lengths taken to frustrate automated blocking, are on another level. To give you some idea of the scale, one of our shared hosting servers has in the last month been averaging over 1.5 million fraudulent requests from 290,000 unique IP addresses per day.

Get the Android app