I benchmarked 5 managed graph databases — and the "obvious" winner changed depending on what I measured

I conducted a benchmark test on five managed graph databases: CognoDB Cloud, Neo4j AuraDB, Memgraph Cloud, FalkorDB Cloud, and ArangoDB Oasis. All databases were deployed in free-tier cloud instances, running identical datasets and queries. The dataset used was SNAP's cit-HepTh citation network, consisting of 27,770 physics papers and 352,807 citation edges, with a simple schema of (:Paper)-[:CITES]-(:Paper).

Five metrics were measured across all platforms: data load speed, traversal times (1/2/3 hops), key lookups, full-graph aggregation, and concurrent load performance. The results were surprising in several ways. Initially, Memgraph appeared to be the clear leader, but full-graph aggregation revealed a different winner. Memgraph led on single-hop queries, but Neo4j AuraDB emerged as the winner for aggregation.

CognoDB and ArangoDB performed significantly slower in aggregation, with times reaching over 1.8 seconds and 4 seconds respectively. The aggregation workload exposed architectural differences between the databases, revealing that a single-query benchmark may not accurately reflect a database's performance under real-world conditions.

Another surprising finding was the performance of ArangoDB under concurrent load. While other platforms showed a 3.6x to 4.1x increase in throughput with increased clients, ArangoDB remained at 15 ops/sec at 10 clients, and only 16 ops/sec at 40 clients. Further investigation revealed that ArangoDB's free-tier Oasis deployment may be hitting connection or protocol limits, making it susceptible to performance issues under load, especially compared to other platforms using binary protocol (Bolt for three of the others, native RESP for FalkorDB).

CognoDB performed relatively well in the benchmark, mid-pack on traversals and lookups, and slower than Memgraph and Neo4j but faster than FalkorDB and ArangoDB. Interestingly, CognoDB supports the same protocol and Cypher as Neo4j, and the connection code was easily adaptable to switch between the two. This provides a practical advantage for teams already using Neo4j who are considering alternative managed graph databases.

However, it is important to note that the benchmarking results were not normalized for hardware or regional deployment, which could impact the overall performance comparisons.