Building a Local Search Index for a 200MB X Archive

The archive arrives as a ZIP file and contains a single file called tweets.js. This file starts with an assignment statement, followed by one very large JSON array. The content of interest is usually a single detail such as a house number, email address, or the name of a hotel from a trip. Reading this data without any setup is challenging and already solved in other ways. However, creating a local index that can be queried quickly is a different approach.

The size of the JSON array is significant; it can be three to five times larger than the original source file due to V8's per-object overhead. Traditional methods of parsing and holding the deduplicated strings in memory hit the default heap ceiling and result in a "heap out of memory" error. Raising the heap limit is not a viable solution.

To overcome this issue, the recommended approach is to stream the structure away, processing each element one at a time and releasing it before moving on to the next. This way, the memory usage scales with the largest individual post rather than the total number of posts. The script begins by importing necessary modules like fs and readline, then opening the source file and creating a write stream for the index in NDJSON format.

The script reads the file line by line, concatenating lines until a complete JSON object is found. It then parses the object and writes the relevant fields to the index file. If the object is split across multiple lines, the script continues accumulating until the complete object is parsed. This process continues until all lines have been processed.

When querying the index, a simple grep command can be used to search for specific terms within the index.ndjson file. This provides a fast and efficient way to search through the entire index. For more advanced querying, the NDJSON index can be loaded into a local full-text engine like DuckDB. This allows for case-insensitive matching, counts, and ranking results based on the date of the post.