Get the latest tech news

The Cost of Being Crawled: LLM Bots and Vercel Image API Pricing

A misconfiguration that might have cost us $7,000

We can't say definitively which bots were downloading images, because we are on the Pro plan on Vercel and no longer have access to logs from Friday. Since both of us were new to managing a large-scale content site (our background is in backends, APIs, and web apps behind auth), we didn't even think about LLM bots. But when we understood that the root cause of the problem lied in the misconfigured image optimization, we decided to keep our site open to all LLM and search engine bots.

Get the Android app