No cloud to lean on: building a reliable data platform on a bank's own servers
Replication, failover, graceful shutdown, autoscaling and alerting usually arrive as managed services. Inside a bank, you build them from Kubernetes, an operator or two and a lot of small decisions. Here is how each one works, with the code. Field notes from moving a transaction reconciliation platform from AWS onto Kubernetes inside banks' data centres The constraints Five words, one goal…
The article discusses the challenges of building a reliable data platform on a bank's own servers, using Kubernetes and avoiding managed services. The goal was to ensure no data is lost, and the system stops no work for more than a few seconds. The platform's functionality was to process file uploads through a web portal, clean the files, load them into an Iceberg data lake using Spark, and reconcile transactions across sources. All communication between services occurred through a message queue.
Key aspects include:
1. Reliability: The system acknowledges work only after it is completed, preventing double lock acquisition and ensuring files are not dropped during shutdown.
2. Fault tolerance: Replicated databases and queues, automatic failover, and failure containment were implemented. If a service fails, it leaves messages waiting in the queue, preventing data loss.
3. High availability: The system aims to minimize user impact by having seconds of downtime for a lost server and zero downtime for upgrades and node maintenance.
4. Scalability: Replica counts follow queue depth, with heavy jobs waiting for available resources instead of failing.
5. Observability: Alerts via email, searchable logs, dashboards, and daily health reports were set up to monitor the system.
The platform's design follows the rule that no work is lost silently, meaning messages are only acknowledged after completion. A message broker like RabbitMQ is used, with quorum queues that drop messages after failed deliveries and dead-letter queues for unrecoverable messages. The platform also includes measures to handle memory issues, such as setting memory limits based on the largest input and monitoring for OOM kills.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
