Mongodb Partitioning
At Whoz , we build a SaaS platform that helps professional services companies manage their talent staffing. At the heart of our product lies a concept called a worklog — a record of time spent by a user on a given activity. Every consultant, every day, on every project, generates worklogs. It sounds simple. And for years, it was. Then the numbers caught up with us. The Problem: A Collection That…
Whoz, a SaaS platform for professional services companies, faced challenges when their MongoDB collection of worklogs grew to 530 million documents, consuming over 32 GB of data. With the growth rate accelerating due to users splitting activities into finer-grained entries, routine operations like backups and restores became increasingly painful.
Three options were explored before settling on application-level partitioning. MongoDB sharding was ruled out due to excessive operational overhead, as worklogs from years ago are rarely queried and would require deploying additional replica sets for high availability. WiredTiger tiered storage was not supported by MongoDB, making it a dead end.
The chosen approach was to split the collection into two separate collections with a fixed cutoff date: one for recent, actively queried worklogs, and another for historical worklogs that are rarely accessed. This solution offers simplicity, with a single $unionWith in a MongoDB view exposing both collections as one. The application remains unaware of the split, eliminating the need for routing logic or dynamic collection names.
While the cutoff date does not move automatically, it provides a manageable starting point for partitioning, with periodic re-partitioning as a future consideration.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
