Debian Code Search: Fast TurboPFor with Go SIMD

In August 2026, the author achieved their long-standing goal of removing the last cgo dependency in Debian Code Search (DCS). This feat became possible thanks to Go's newly introduced SIMD support, which allowed implementing the TurboPFor integer compression format more efficiently than the reference implementation. DCS is a search engine that enables searching all open-source source code within Debian using literal search expressions or regular expression search queries.

It relies on an inverted index to map terms to documents containing those terms (posting lists). The index is typically represented by id lists for each document to minimize memory usage. When searching, decoding these id lists quickly is crucial. Since 2019, DCS has utilized an on-disk positional index, which significantly improves query performance for literal queries.

The efficient encoding and decoding of TurboPFor format rely heavily on the optimized C TurboPFor library's decoder. However, the author aimed to eliminate the C TurboPFor dependency and develop a native Go version instead. With the release of Go 1.26 in February 2026, access to a new experimental simd/archsimd package was granted, providing architecture-specific SIMD operations.

By leveraging this package, the author successfully implemented a native Go encoder for TurboPFor, achieving performance comparable to the C TurboPFor dependency. In addition, the author discovered a technique called positional popcount, which further improved the encoder's speed by 2x. While the Go implementation is not as fast as the C version, the result was still impressive, achieving a performance improvement that surpassed DCS's previous state.