Get the latest tech news

Google and Microsoft among those signing up to OpenAI's plan for a new website crawling standard — including opting-out of internet scraping

One robots.txt line could soon stop scraping

None

Get the Android app