Pipeline Deployment & Container Lifecycle with Background Polling: Building Debezium's Host Path
I'm contributing to the Debezium Platform as part of Google Summer of Code 2026. In my previous post, I covered the host provisioning engine — how the system discovers SSH hosts and installs Docker on them. This post covers what happens next: actually deploying pipelines to those hosts and keeping them alive. Where We Left Off From my previous POST , the platform could watch ~/.ssh/config ,…
This report outlines the process of deploying pipelines to remote hosts using Debezium's Host Path, as well as continuously monitoring container health and detecting any tampering with configuration files on disk. The key aspects covered include the container runtime abstraction, enterprise clean architecture refactorings, and background status polling.
The container runtime abstraction was designed as an interface called HostContainerRuntime, which provides methods for deploying, undeploying, stopping, starting, and retrieving logs of containers on remote hosts. This interface enables the use of different implementations, such as AnsibleContainerRuntime and AgentContainerRuntime, without requiring changes to the controller code. The latter uses CDI to inject a different implementation depending on the deployment context.
The refactorings performed during code review to improve the overall enterprise clean architecture of the system included the following actions:
1. JPA Entity Hiding & Boundary Decoupling: This involved creating lightweight, immutable Domain Records (Deployment and HostStatusReference) to replace ORM entities that were previously returned from the HostDeploymentService. This approach addressed issues such as Hibernate Dirty Checking, LazyInitializationException, and tight coupling between controllers and database column definitions.
2. Concurrency-Safe Host Selection: When a pipeline is created, the system must determine which host it should run on and which port to use. This is accomplished through pessimistic database locking, ensuring that only one transaction can access a given set of rows at a time. This prevents race conditions where two pipelines might end up on the same host and port.
3. Pluggable Deploy Strategy: The DeployStrategy interface allows for multiple strategies to be implemented for selecting the best host for a pipeline. The first implementation, LeastLoadedDeployStrategy, picks the host with the fewest active containers, ensuring optimal resource utilization. The live COUNT(*) query runs inside the pessimistic lock so that the count is always current.
Finally, background status polling is implemented to continuously monitor container health and detect any tampering with configuration files on disk. This ensures that pipelines remain operational and that any issues can be quickly identified and addressed.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
