How much of your ML workflow is still held together manually?
I've been working on a DataOps/MLOps product called Datryc, and before we go much further with it, I want to challenge some of the assumptions we're making. The idea came from a fairly simple observation: getting from raw data to a model running somewhere involves much more than just training the model. You may need to: connect to different data sources; validate and clean the data; build…
Writing about the Datryc DataOps/MLOps product, the author expresses concern about the manual processes still involved in moving from raw data to a running model. While several excellent tools exist to handle specific aspects of this pipeline, the author wants to understand the integration challenges that arise between these components.
The Datryc team is experimenting with a common API/CLI to tie these workflows together, keeping the underlying modules modular. They utilize asynchronous workers and Kafka to decouple pipeline execution from the API layer. However, the author warns that it's a mistake to focus on infrastructure before confirming that the workflow itself is a real pain point for users.
They seek insights from professionals working with production data and machine learning systems on what they perceive as the most labor-intensive or unnecessary manual tasks in their current Data/ML processes. The author is specifically interested in various aspects such as data preparation and quality, migrating pipelines between environments, linking different ML/data tools, model deployment, monitoring, ensuring reproducibility, and integrating ML workflows into existing CI/CD pipelines. They also invite those using a fully-functioning current stack to share what solutions they employ.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
