The Pipeline Worked. Then the Research Outgrew It.

A year ago, I was developing a terminal-based workflow manager called Glyph.Flow, which served as a learning project. As the workflows grew more realistic, I built a data pipeline for my Master's thesis to create and process a cross-national research database from various sources. The pipeline proved effective, but as my research progressed to a PhD, the question arose: what would be needed for long-term research infrastructure?

The original system was designed to finish a specific task; however, my PhD required a research infrastructure capable of adapting to evolving research questions, new datasets, and methodological changes. Questions like "Does it work?" became inadequate, replaced by inquiries such as "Can I extend it without breaking old processing steps?" and "Can I rerun only the parts affected by a change?"

The success of working software can create a trap where the architecture is deemed validated, but this only confirms the system's current capabilities. In research software, the evolving code and research process demand a focus on dependency, provenance, validation, maintainability, and architecture.

As I shifted my thinking from a research product to a research process, I began to consider implementation details as first-class design problems. Addressing validation, transformations, dependencies, and information storage became crucial. The challenge lies in finding a balance between too little flexibility (leading to constant refactoring) and too much (wasting time on building an overly general framework).

Ultimately, I aim to design a smaller, safer, and more transparent infrastructure that enables understanding the research process and facilitates future extensions.