Your Orchestrator Is an Operating Model, Not a Feature List

Data orchestration has regained relevance, though its complexity remains undiminished. In July, Prefect announced the acquisition of Dagster Labs, stating that Dagster will retain its name, support, and open-source license without demanding any migration from existing customers. In September, Kestra released version 2.0, featuring a revamped engine, distinct control and data planes, remote workers, and flows accessible to agents as tools.

Consequently, the question resurfaced: which orchestrator should we adopt? The discussion on Reddit featured responses from various stakeholders, ranging from recommending continuation with Dagster to switching to Airflow due to its established presence, choosing a managed service, or opting for a smaller solution for lean teams.

Some vendors also stepped forward to promote their respective products, given the social nature of certain distributed systems laws.

I believe the focus should be on the comparison of orchestrators rather than a simple feature list. An orchestrator essentially defines how a team manages work. The syntax and user interface matter, but the critical decision lies in determining who establishes workflows, where code executes, how failures are handled, and which platform boundaries are enforced.

The Directed Acyclic Graph (DAG) is merely the simplest aspect. Most tools can depict this process: Extract | v Transform | v Validate | v Publish. A demonstration typically concludes once those four stages are validated. However, true production begins once the second box runs for five hours, the third box encounters data issues post-publish, and a team needs to backfill data for Tuesday without executing Wednesday's processes repeatedly.

This is where orchestrators diverge significantly from feature matrices, which often fail to capture these nuances.

When selecting an orchestrator, begin by establishing the ownership boundary. Platform teams should manage the control plane, authentication and secrets, worker environments, deployment path, shared monitoring, while data teams should handle workflow definitions, transformation code, data-quality rules, schedules and dependencies, domain runbooks.

This model clarifies the questions at hand. If each data team requires the platform team to install a Python dependency, self-service is primarily a branding exercise. Conversely, if any workflow author can execute arbitrary code with shared production credentials, self-service has likely taken a step too far. The orchestrator should facilitate the enforcement of the desired boundary.

Kestra 2.0 stands out in this context due to its release notes mentioning workers connecting to the control plane via an outbound gRPC stream. User code runs in the data plane, allowing workers to reside in different regions, clouds, or outbound-only networks. This signifies more than just an engine detail; it enables an operating model where a central team oversees orchestration while execution occurs near the workload.

However, it also entails migration work. The 2.0 release involves upgrading from 1.3.x, alters the default behavior of unmatched worker routing from waiting to failing, and necessitates an upgrade through 1.3.x. These factors should be considered in the decision-making process, as an orchestrator is a long-term operational dependency, not a library we can casually replace on a Friday afternoon.

When deciding on an orchestrator, it is essential to consider what the system aims to organize. The task-versus-asset distinction is another operating choice. A task-oriented model questions, "What should run next?" It aligns naturally with jobs, scripts, APIs, and infrastructure operations, typically appealing to teams managing mixed workloads.

Conversely, an asset-oriented model asks, "What data should exist, and what does it depend on?" This approach streamlines partitions, lineage, freshness, and backfills for data-intensive platforms. Neither model is inherently superior. The crucial question is what engineers primarily debug. If incidents frequently manifest as "this job did not run," task state may be more beneficial.

If concerns revolve around missing customer data from the previous day's partition, asset state could prove more useful. If the same platform oversees infrastructure automation, ML training, and API workflows, a model heavily reliant on data-specific abstractions may become cumbersome. Choose the model that aligns with the failure language and operational context of your team.

The acquisition of Prefect by Dagster represents a significant signal, even though it does not automatically render a running Dagster deployment unsafe. The announcement assures continued support, maintains the existing name, and preserves the open-source license. However, this fact alone does not dictate a migration path. To address potential future convergence, consider pinning the current version, testing upgrades in a representative environment, documenting APIs, metadata, and deployment assumptions to mitigate lock-in, exporting workflow and run metadata for audit or migration, and defining triggering events that would prompt a reevaluation.

These steps constitute essential engineering practices. Regardless of the orchestrator, project ownership, cloud pricing, managed features, and open-source project sustainability are all subject to change. An exit strategy is integral to the architecture. Conduct a failure-focused proof of concept to evaluate a small workflow, intentionally introduce complications, and explore the platform's response.

By approaching decisions with a proactive, failure-oriented mindset, you can better navigate the complexities of orchestrator selection and ensure the long-term viability of your data workflows.