Get the latest tech news

Can we RAG the whole web?

, or Retrieval-Augmented Generation, is a method where a language model such as ChatGPT first searches for useful information in a large database and then uses this information to improve its responses. This article implies some prior knowledge on vector embeddings.

The XML sitemap that I mentioned earlier is a protocol that was invented by Google back in 2005 and is widely adopted among the web as a way to indicate urls to crawl. SQLite keeps an entire relational database contained within a single file, allowing for straightforward deployment and minimal setup while still delivering powerful features natively or via extensions. Typically, RAG is implemented with a vector similarity search algorithm, and thankfully, SQLite has an extension written by Alex Garcia, called sqlite-vss, that does exactly that for us.

Get the Android app