Get the latest tech news
NewsCatcher (YC S22) Is Hiring Web Scraping/Crawling Engineers
& Scraping Engineer 🌟 We seek a highly skilled and motivated Web Crawling & Scraping Engineer. We crawl over 100k news websites daily and we are looking for someone who is passioned about Web Crawling the same way we are.
Curious to explore new ways of handling difficult website cases and automating our crawling techniques. Add an automatic way of fixing non-working crawlers Provide metrics on website coverage Data Pipeline: Design, construct, test and maintain robust, reliable, and data pipeline infrastructure. Introduce a new automatic way of crawling a website that does not work with existing techniques Come up with an idea on how to verify why a specific crawler stopped working and fix it automatically Use LLM methods to improve crawling methods
Or read this on Hacker News
