Get the latest tech news

Websites and web developers mostly don't care about client-side problems


You're probably reading this page because you've attempted to access some part of my blog (Wandering Thoughts) or CSpace, the wiki thing it's part of. Unfortunately whatever you're using to do so has a HTTP User-Agent header value that is too generic or otherwise excessively suspicious.

Unfortunately whatever you're using to do so has a HTTP User-Agent header value that is too generic or otherwise excessively suspicious. Unfortunately, as of early 2025 there's a plague of high volume crawlers (apparently in part to gather data for LLM training) that behave like this. To reduce the load on Wandering Thoughts I'm experimenting with (attempting to) block all of them, and you've run into this.

Get the Android app

Or read this on Hacker News

Read more on:

Photo of Websites

Websites

Photo of problems

problems

Photo of Web developers

Web developers

Related news:

News photo

AI crawlers, fetchers are blowing up websites; Meta, OpenAI are worst offenders

News photo

AI crawlers and fetchers are blowing up websites, with Meta and OpenAI the worst offenders

News photo

Phison takes legal action over falsified 'leaked' document on Windows SSD issues — says it continues to investigate reports of problems