I got tired of surprise AWS bills, so I open-sourced a Multi-Cloud FinOps Engine.

Many engineers find cloud billing to be a confusing and complex process, particularly when managing multiple cloud providers like AWS, GCP, and Azure. This can create the unpleasant situation of realizing late that a neglected database cluster has been charged $50 per day during the month, even after the bill has been generated. Frustrated with this issue, the author decided to create a solution that would simplify the process of identifying unnecessary cloud expenses.

The Multi-Cloud FinOps Engine is a Python-based FastAPI engine that connects to the billing data of different cloud providers. It then normalizes this data into the FOCUS 1.0 standard, which is a unified format. The engine utilizes local machine learning techniques to detect cost anomalies, providing a single API endpoint to access multiple cloud billing data in a standardized format.

Key features of the engine include:

1. No External SaaS: The engine is designed to be self-hosted, ensuring that sensitive billing data stays within the user's infrastructure and doesn't get sent to a third-party service for analysis.

2. True Multi-Cloud Support: The engine supports AWS Cost Explorer, GCP BigQuery Billing Exports, and Azure Cost Management. It normalizes the billing data into the FOCUS 1.0 standard, which simplifies the process of answering questions about cloud spending.

3. FOCUS 1.0 Normalization: One of the biggest challenges with multi-cloud FinOps is the varying billing schemas, dimensions, terminology, and metadata of different cloud providers. The engine resolves this issue by normalizing all data into the FOCUS 1.0 standard.

4. Zero Memory Spikes During Large Exports: Billing datasets can be massive, which can cause issues with memory usage when dealing with millions of rows. The engine addresses this by implementing FastAPI StreamingResponse generators, enabling the API to stream data progressively without causing a spike in RAM usage.

5. Non-Blocking Machine Learning: Running Scikit-Learn models such as Isolation Forest against large datasets can be expensive and potentially overload the FastAPI event loop. The engine tackles this by routing the machine learning workloads, external cloud SDK calls, and other blocking operations through ThreadPools.

The architecture of the Multi-Cloud FinOps Engine consists of an API client that interacts with the FastAPI API. The FastAPI API uses Pydantic v2, Pandas, Scikit-Learn (for Isolation Forest), and Docker, Kubernetes, and KEDA for infrastructure setup. The entire system is containerized and designed to be deployed in a Kubernetes environment. The codebase is open-source and available on GitHub, inviting feedback and improvements from the community.