Jepsen: Orkes Conductor 5.3.2

Orkes Conductor is a workflow orchestration platform that runs user-defined tasks and routes data between them. In testing versions 5.3.2 through 5.5.0-rc2, unexpected concurrent task execution was observed, which could significantly delay workflow execution. Additionally, the platform exhibited four minor issues with startWorkflow errors or null IDs, and three serious problems that caused Conductor to permanently abandon or forget workflow executions, or mark them as completed when their tasks were never run.

Orkes has since released version 5.5.6, addressing all serious safety issues and reducing concurrent execution, while two startWorkflow errors remain under investigation. Delayed execution aligns with Conductor's design parameters and should not be addressed. This research, funded by Orkes Inc., was conducted ethically in accordance with the Jepsen policy.

A workflow orchestrator is a software system that executes computations made up of small tasks, which are connected in a workflow to route outputs to inputs, determine scheduling, and manage workflow and task completion. Conductor, originally built by Netflix and open-sourced in 2016, is maintained by the Conductor OSS Foundation, led by Orkes, which also offers a hosted version called Orkes Conductor.

In Conductor, a workflow definition is a JSON structure defining tasks, their order, inputs, outputs, and other parameters. Each workflow definition has a name and version, and each execution is identified by a unique workflow ID. Tasks may be attempted multiple times during an execution, with unique execution IDs assigned to each attempt.

Conductor's five major components include servers, a database, a queue, workers, and custom workers written by users using SDKs for various languages. All components can be scaled independently, and Conductor guarantees at least once message delivery and eventual convergence of workflows on a terminal status.