Building a Logging Platform

Log management poses unique challenges for distributed systems. While applications can effortlessly create logs, few can effectively comprehend them at scale. Printing a thousand error messages is easy, but handling ten million becomes interesting when your application runs across many machines. This necessitates addressing questions about log storage, search, performance, handling out-of-order logs, noisy services, indexing various data formats, aggregating logs, handling machine failures, automatic log expiration, and balancing fast writes with slow queries.

The solution lies in building a logging platform that functions as a distributed architecture designed to handle vast streams of semi-structured events. A logging platform essentially turns a massive flow of semi-structured events into searchable, filterable, aggregable, retained, and distributable data.

To illustrate, consider an e-commerce platform with services like API Server, Payment Service, Order Service, Inventory Service, Notification Service, Worker Nodes, and a Database. Each of these services generates logs. For instance, an error from the Payment Service might look like this: { "timestamp": "2026-09-18T14:32:51Z", "level": "ERROR", "service": "payment", "host": "payment-03", "message": "Payment failed", "order_id": "9812", "status": "402" }.

Another example is an inventory update from the Inventory Service: { "timestamp": "2026-09-18T14:32:52Z", "level": "INFO", "service": "inventory", "message": "Stock updated", "product_id": "P102", "quantity": 48 }. Developers would want to ask questions such as finding all ERROR logs from the Payment Service during the last hour, displaying all requests with status code 500, counting payment failures per minute, finding logs containing the word "timeout", or showing the top 20 services producing errors.

The architecture of a logging platform divides responsibilities into distinct layers. Applications generate logs, Log Collectors gather them, the Ingestion Layer validates and normalizes them, Buffer/Queue absorbs bursts, Indexing Nodes transform logs into searchable structures, Storage Nodes persist them, Query Coordinator distributes searches, and finally, the API/ UI exposes everything to humans and applications. This separation allows the system to scale effectively.

At the core of this system is the Log Event, defined as a struct LogEvent { timestamp : i64, level : String, service : String, host : String, message : String, fields : Map<String, Value> }. Here, the fields field is particularly interesting because modern logs are rarely flat. They may contain nested structures like { "user": { "id": 42, "country": "ZM" }, "request": { "method": "POST", "path": "/api/payments" } }.

Users might search for specific fields, such as user.country = "ZM" or request.method = "POST". This leads to a critical design decision: while storing logs as raw JSON preserves the original structure, direct search of raw JSON might not be efficient.