Get the latest tech news
Drawbacks and solutions for the Meilisearch document indexer
In this blog post, we explore the enhancements needed for Meilisearch's document indexer. We'll discuss the current indexing engine, its drawbacks, and new techniques to optimize performance. With insights on parallel processing, memory management, and efficient data handling, we'll draft a robust new indexer to better handle large datasets and frequent updates. Join us for a deep dive into making Meilisearch's indexing faster, more efficient, and scalable.
Users send their documents through HTTP routes, and Meilisearch accepts them asynchronously, converts them into grenad JSON files and registers them in the index scheduler. To fix this, we can process the documents in smaller chunks across different transactions, and since the database is empty when loading a dump, using the MDB_APPEND option helps add entries and cut down on dirty pages. Currently, this function is applied to all documents in parallel, resulting in a single grenad JSON file before continuing with the standard indexing process.
Or read this on Hacker News
