I set out to climb "graph engineering." It was the ladder I'd been standing on for 25 years.

The author embarked on a journey to explore the field of graph engineering, which they perceived as the next step in their 25-year career in enterprise infrastructure. The concept of graph engineering revolves around structuring knowledge as entities and typed relationships instead of flat documents, enabling systems to traverse connections instead of just keyword-matching text.

This approach opens up new possibilities for asking more complex questions, such as "what is two hops downstream of X, and what breaks if it goes away?"

The author, who spent their career in various areas of enterprise infrastructure, including virtualization, storage, cloud, and data centers, noticed a familiar pattern in their work. They described this as dependency maps, impact analysis, entity-relationship modeling, failure domains, and org charts. These are all techniques used to understand dependencies and relationships within complex systems, and they draw a parallel between these concepts and graph engineering.

While the author acknowledges that the techniques used in graph engineering are not entirely new, they do point out that the methods of building and maintaining these graphs have changed significantly. The advent of large language models has enabled machines to extract entities and relationships from unstructured text at scale, making it possible for graphs to be built and maintained by machines rather than armies of analysts. This is a significant shift from the traditional approach.

The author also highlights the role of GraphRAG (graph retrieval augmented generation) in improving multi-hop questions that plain retrieval struggles with. On large corpora, such as millions of documents or codebases, GraphRAG has been shown to be effective in answering complex questions. However, the author also points out that the technique is scale-gated, meaning that it only becomes effective at scales that are difficult to achieve with a small team or limited resources.

In light of these findings, the author concludes that the concept of graph engineering is not entirely new, but rather a rebranding of existing techniques. The real novelty lies in the ability to build and maintain graphs using large language models, and in the potential of GraphRAG to improve multi-hop question answering on large datasets. The author also notes that these techniques may not be as effective at their current scale, emphasizing that the scale gate swings both ways.