Get the latest tech news
Stop Scraping My Git Forge
my Git Forge 2024-06-03 Amazonbot, please. It's too much.
Organising thought or recalling specific terminology/techniques could sometimes be tricky, and being able to rubber duck with it proved handy as a springboard to go do "serious" research with a search engine or technical documents. But I saw a massive uptick in the amount of data (many gigabytes over a few days) that a certain Amazonbot( Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/600.2.5 (KHTML, like Gecko) Version/8.0.2 Safari/600.2.5 (Amazonbot/0.1; +https://developer.amazon.com/support/amazonbot)) was pulling from git.gmem.ca and decided to take a slightly closer look. Quicky checking the link included in the user agent, it appears it's used for Alexa related queries, but Cloudflare Radar lists it as an "AI Crawler", which makes me suspect it's also feeding into whatever machine learning models they're building at AWS.
Or read this on Hacker News