Building an investing knowledge graph, part 6: if you're merging data from more than one source, you'll hit this

Building an investing knowledge graph, part 6: merging data from multiple sources can be challenging. The author discusses two conversations they had regarding merging data from different sources. The first conversation involved building a financial news aggregation system, while the second was about KYC screening. The KYC conversation posed unique challenges, as the error tolerance was significantly different than in the financial news case.

In the KYC process, a false negative could mean that an entity that should have been flagged clears a check, which could have severe downstream consequences. The author explains that the core architecture, such as the pairwise scoring, registry with a stable and pending boundary, and review layer for uncertain pairs, remains the same across domains.

However, the model, threshold, and blocking strategy would need to be adjusted for each specific domain. For the financial news aggregation, cross-language blocking was relatively straightforward due to the use of string normalization and phonetic blocking. In contrast, Chinese-English matching required a more sophisticated approach, such as alias generation during ingestion to produce language variants and store them as additional candidates for blocking.

The author emphasizes that the registry architecture is more transferable than the specific model and threshold values used. They advise budgeting for registry maintenance, as the ongoing cost lies in managing accumulated state rather than running inference. Additionally, the blocking step can be underestimated, as the model may not see pairs that did not make it through blocking.

The author recommends using Splink for probabilistic lifting, a well-maintained open-source library, before building custom solutions. The investing knowledge graph is still running, with the entity resolver live on Railway as an ER API. The author plans to expand access based on early user feedback and provides a form at hannune.ai for those interested in trying the service.