{
  "id": 18195475,
  "title": "From YAML validator to event-driven projects: 14 releases of Hydra ETL in six weeks",
  "url": "https://urgent.news/2026/11/04/from-yaml-validator-to-event-driven-projects-14-releases-of-hydra-etl",
  "topic": "tech",
  "section": "Tech",
  "published": "2026-11-04T22:34:34.000Z",
  "source": {
    "name": "Dev.to",
    "slug": "dev-to",
    "url": "https://dev.to/bejaouibechir/from-yaml-validator-to-event-driven-projects-14-releases-of-hydra-etl-in-six-weeks-3lgc"
  },
  "original_language": "en",
  "account": "Hydra ETL has undergone significant evolution over the past six weeks, with 14 releases since September. At its core, a Hydra ETL job consists of four YAML files: sources.yaml, transformations.yaml, destinations.yaml, and pipeline.yaml. These files are only executed if their manifests are valid, ensuring data integrity from the start.\n\nIn version 0.11.4 to 0.12.3, Hydra ETL introduced a project-based structure, where multiple jobs can be orchestrated in a defined order. The hdrctl init command now scaffolds a new project, including jobs, workflows, parameters, and environments. The hdrctl run command checks the entire project, listing any issues before execution. Additionally, Studio no longer overwrites externally edited files, ensuring consistency between the visual editor and the project files.\n\nVersion 0.13 to 0.14 focused on preventing partial data loads in case of failures. All-or-nothing load per job ensures that if a job fails halfway, the destination remains unchanged. Rollback functionality allows failed workflows to undo their steps, and fallback workflows can take over under certain conditions. Timeouts can be set per workflow to prevent indefinite execution.\n\nThe subsequent versions 0.15 to 0.16 introduced enhancements for handling large datasets. While pandas remains the default engine, Hydra ETL can now leverage pyarrow and DuckDB to process CSV files with up to 20 million rows, executing aggregations, sorts, and joins efficiently. Engine selection per job is also possible.\n\nVersion 0.17 to 0.18 added improved security and environment management. Secrets are now masked in logs, errors, and tracebacks, and can be encrypted using SOPS with cloud KMS support. Projects can now have multiple environments with global variables, parameter values, and secrets specific to each environment. hdrctl env, hdrctl param, and hdrctl secret commands facilitate managing these configurations.\n\nVersion 0.19, released on November 4 (the author's birthday), introduced project triggers, enabling projects to start automatically based on various events. The triggers.yaml file defines triggers such as cron jobs, file changes, webhook events, and the completion of another project. This feature allows seamless automation and integration of Hydra ETL projects with external systems. Additional features introduced in this release include sub-projects, cycle detection, and result sharing between projects.",
  "summary": "At the end of September, Hydra ETL did one thing well: you described a data job in four YAML files, hdrctl validate checked them before any data was read, and the same job ran from the terminal, a REST API or a visual editor. Six weeks and 14 releases later, a Hydra ETL project can start itself when a file lands in a folder, roll back what it wrote when something fails, aggregate 20 million rows…",
  "key_points": [
    "Hydra ETL evolves through 14 releases in six weeks",
    "Introduces project-based structure with jobs, workflows, and environments",
    "Version 0.19 adds project triggers for automated execution"
  ],
  "editors_take": null,
  "illustration": null,
  "coverage": {
    "outlets": 1,
    "also_reported_by": []
  },
  "ai_generated": true,
  "disclaimer": "Summaries, key points and the editor’s take are written by software from other outlets’ reporting and may contain errors — always check the linked original."
}