Building certgrep.sh: a free certificate transparency search engine

Certificate transparency offers a valuable dataset for security professionals. Every certificate issued by a trusted certificate authority is recorded in an append-only log, often before the certificate is even used. This provides early indicators of malicious infrastructure. However, managing access to this dataset can be challenging.

Third-party search tools may be unreliable, slow, or expensive. To address these issues, Have I Been Squatted developed certgrep.sh, a free, open-source tool for searching certificates with full regular expression support.

Initially, certgrep.sh was built as an internal tool to handle the high query volume needed for their detection pipeline. They faced issues with existing public search tools and costly commercial APIs. Their approach revolved around the fact that certificate metadata is secondary to domain occurrences. They focused on efficient domain name lookups, particularly using regular expressions for typosquatting and phishing patterns.

The solution involved using finite state transducers (FSTs), specifically the Rust fst crate. FSTs are deterministic automata that efficiently handle lookups by key length, store data as a flat byte sequence, and allow for fast pattern matching using regular expressions. By compiling regular expressions to DFAs and intersecting them with the FST, certgrep.sh can quickly determine if a domain pattern matches any certificate in the logs.

Domain names, being short and highly repetitive, compress well, making the FST an ideal solution. Once compiled, the FST is immutable, ensuring the data remains consistent. This architecture allows certgrep.sh to provide fast, free access to certificate transparency data while minimizing storage costs and operational overhead.