How I Automated 10M+ Network KPI Records a Day with Python and Apache Airflow
Originally published on pmpdilshan.github.io . Every hour, a mobile network generates performance counters for every cell: traffic, drops, handovers, throughput, congestion. At Mobitel that adds up to more than 10 million KPI records a day. For a long time, turning that into reports meant exporting files, merging them in Excel and rebuilding the same charts every morning. So I built pipelines in…
Every hour, mobile networks generate performance counters for each cell, including traffic, drops, handovers, throughput and congestion. This results in over 10 million KPI records daily for Mobitel alone. Traditionally, turning these raw data points into reports involved manual export, merging and charting, leading to a massive 40% reduction in effort after implementing automated pipelines using Python and Apache Airflow.
The pipeline begins with extraction, pulling counter exports from network management systems on a schedule. Next, validation checks file arrival, row counts and data completeness. Transformation then standardizes names, calculates KPIs and aggregates results by cell, cluster and region. The final step is loading the data into a queryable database, followed by refreshing dashboards and setting alerts for any issues.
Critical lessons from this automation journey emphasize the importance of validation, making tasks safe to rerun, defining KPIs centrally and maintaining thorough logging for debugging. By automating the laborious reporting process, Pathum Dilshan, an RF & Network Planning Engineer at Mobitel and MSc Data Science & AI student, has freed up valuable time for optimization work. He encourages others to prioritize building the automated reporting system itself rather than solely focusing on the reports themselves.
Rewritten by urgent.news; may contain errors — check the original.