The Workflow That Passed Every Test and Failed on Real Data: Production Payload Shapes in n8n

This article from the "build-in-public" series discusses the challenges faced when running large language model (LLM) workflows in n8n in a production environment. The author shares their experience of how the workflow initially passed all their test cases, but failed when confronted with real data. The failure was not due to any bugs in the logic, but rather due to the shape of the data that differed from what was tested.

The author identifies three main issues that caused the failure: fields that are always present in test data but sometimes absent in production; type drift where the same field arrives in different types; and scale and content extremes that were not accounted for in test data. To address these issues, the author recommends implementing a validation step that sits between the trigger and the rest of the workflow.

This validation step checks for required fields, normalizes the data, and quarantines any data that cannot be fixed. By doing so, the workflow admits what it does not know and allows a human to take action, rather than failing silently or producing incorrect results. The author emphasizes the importance of testing workflows with production data to ensure they can handle unexpected shapes and content.

They suggest running the workflow against a dataset designed to expose its weaknesses, checking for counts of processed, quarantined, and dropped rows, and asserting on the results rather than just the success of the workflow. By adopting these practices, the author argues that workflows can better handle real-world data and avoid producing wrong results.