Get the latest tech news

Deep Bug

The project has been haunted by a mysterious bug since sometime February. It relates to the code that constructs the index, particularly the code that merges partial indices. In short the search engine constucts the reverse index through successive merging of smaller indices, which reduces the overall memory requirement. You can conceptualize the revese index itself as two files, one with offset pointers into another file, which has sorted numbers. This code runs after each partition finishes crawling and processing its data, and has a run time of about 4 hours.

In short the search engine constucts the reverse index through successive merging of smaller indices, which reduces the overall memory requirement. The index construction does all of its work in the bermuda triangle of 32 bit errors that is 1-32 GB file size range, so it really does seem like a very probable suspect. The JVM is at the top of the suspect list since one of the things that had in fact changed since the bug appeared was that the project migrated over to graalvm from openjdk.

Get the Android app