Kestra 2.0: a new engine, workers anywhere, and still Apache 2.0
In February 2022 I published my first post here, called Kestra, infinitely scalable open source orchestration and scheduling platform . I described an orchestrator using Kafka as both queue and database, Elasticsearch as the repository behind the UI, and I was proud of the 350,000 executions a month it was running at Leroy Merlin. Two things about that post did not survive contact with reality.…
In April 2022, Kestra, an open source orchestration and scheduling platform, was first introduced. The initial version relied on Kafka as both a queue and a database, with Elasticsearch serving as the backend repository. However, this architecture proved to be too heavy for practical use, leading to a rewrite of the engine four years later.
The most common feedback received was the need for a simpler deployment process, which was addressed by replacing the Kafka and Elasticsearch clusters with a JDBC backend running on a single Postgres or MySQL server. This change allowed the platform to scale infinitely without the need for heavy infrastructure.
Other updates in the 2.0 release include the introduction of subflows and fan-out over files, enabling larger pipelines to be broken down into smaller, more manageable parts. Task runners were added to allow the same script task to run as a Docker container, local process, or Kubernetes pod, simply by changing one property. Git sync, a browser editor, CLI, and GitHub Actions were also introduced for deploying flows from CI.
The engine rewrite in 2.0 aimed to address the limitations of the previous architecture. By splitting Kestra into a control plane and a data plane, workers can now execute code independently of the central database. Workers establish a persistent gRPC stream with the worker controller, eliminating the need for database credentials and allowing them to run in various environments, such as different clouds, regions, or on-premise setups.
Additionally, internal debt was tackled by decoupling the queue and repository, allowing users to choose the appropriate components independently. The Kafka Streams engine was removed, and workers now communicate through a single, encrypted gRPC channel.
The 2.0 release also brought improvements to the user experience, such as the introduction of a canvas UI and drafts functionality. The AI Agent task allows users to integrate with Ollama on their local hardware, and the Copilot can be disabled if desired. The new UI syncs with the underlying YAML in the Git repository, ensuring that any changes made in the UI are reflected in the code.
Throughout the development process, the declarative YAML syntax, support for any language, and the open-source nature of the platform have remained unchanged. The license has also been maintained as Apache 2.0, ensuring that users can freely use and modify the software.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
