I Built a CI/CD Platform From Scratch — From PipelineOS to OdinRun

The genesis of OdinRun began as an experiment called PipelineOS, an endeavor to construct a CI/CD execution engine from the ground up. Initially, the runner consisted of a single monolithic codebase comprising 810 lines of code, handling everything from API communication to Docker container lifecycle management, log streaming, metric collection, retries, auto-remediation, and task orchestration.

This rudimentary design led to numerous reliability issues such as Docker attach stream corruption, unenforced stage timeouts, zombie containers during SIGTERM shutdowns, and flaky runner registration logic.

The project's first major milestone, PIP-33, was achieved by decomposing the monolithic runner into distinct, single-responsibility components: Container Runner for managing the Docker daemon lifecycle, Stage Runner for handling stage semantics, API Client for communicating with the control plane, and Executor for coordinating the lower-level runners. This restructuring significantly improved testability, isolation, and predictable error handling.

While the new design brought substantial improvements, it also exposed fundamental flaws that had been overlooked. For instance, the original runner treated the raw Docker stdout / stderr multiplexed stream as plain text, corrupting log streams with binary headers. OdinRun now explicitly demultiplexes the Docker stream into separate, clean channels.

Additionally, enforcing stage timeouts became crucial to prevent runaway steps from consuming compute resources. Graceful container cleanup was also implemented, ensuring that active Docker containers are not abandoned during termination signals.

As the system evolved, the once synchronous HTTP REST calls for stage execution and state changes became a bottleneck. This led to the introduction of an in-process Domain Event Architecture, where the runner emits strongly-typed domain events to an in-process bus, allowing independent consumers to subscribe to these events asynchronously.

This change made the execution engine less dependent on external HTTP endpoints, enabling the addition of new capabilities like live streaming and remediation without modifying the core execution loop.

To ensure data durability and reliability, OdinRun adopted pluggable database persistence abstractions (supporting both SQLite and MongoDB) and a durable log and artifact storage abstraction, allowing for seamless integration with various storage solutions like local NVMe drives, AWS S3, or MinIO. This persistence layer provided a clean separation between the core execution engine and the storage backend, ensuring that build history persisted even when nodes restarted.

A significant milestone, PIP-35, was the introduction of rich telemetry metrics directly from the Docker daemon during stage runs. OdinRun started capturing detailed telemetry information, including stage lifecycle state transitions, CPU utilization percentages, and other runtime metrics. This transparency provided valuable insights into the execution process, enabling better monitoring, debugging, and optimization of the CI/CD pipeline.

Through these iterative improvements and architectural transformations, OdinRun evolved from a simple, monolithic runner into a robust, observable, self-healing, and AI-assisted CI/CD platform. The journey highlighted the importance of building reliable execution engines, fixing fundamental issues, and adopting modern architectural patterns to create a scalable, maintainable, and user-friendly CI/CD solution.