How Fan-Out/Fan-In Works in Dapr Workflow
Introduction You have a list of items to process, and each item takes a while. Running them one after another wastes time, and running them in parallel on threads loses everything when the process dies halfway. Fan-out/fan-in in Dapr Workflow solves both. You schedule a batch of activities at once, wait for all of them with when_all , and combine the results. Each finished activity is recorded in…
Fan-out/fan-in is a workflow pattern in Dapr that allows processing of a large list of items in parallel, while ensuring that all results are recorded in the workflow's history for recovery after crashes. By scheduling a batch of activities at once, waiting for their completion with when_all, and combining the results, fan-out/fan-in optimizes time consumption and handles process failures efficiently.
The Python SDK 1.19 of the Dapr package provides built-in support for this pattern. Install it using pip install dapr[workflow]. The pattern's key steps involve getting a list of items, starting a process_item activity for each item, waiting for all tasks to complete with when_all, and finally aggregating the results using a save_total activity.
In case of a crash during the process, the workflow resumes from the point where it left off, skipping the completed activities and rescheduling only the unfinished ones. However, when implementing fan-out/fan-in, it is essential to consider certain limitations, such as the dispatch size cap of 4 MiB by default, which may lead to workflow stalls if exceeded.
To avoid this, ensure that returned results are small and consider splitting large fan-outs into child workflows or scheduling items in chunks. Additionally, plan for failures by setting appropriate retry policies or handling errors within activities. Fan-out/fan-in is just one of the six workflow patterns provided by Dapr, alongside task chaining, async HTTP APIs, monitor, external system interaction, and compensation.
Written by urgent.news from Dev.to's reporting — not their text. Machine-written — may contain errors; check the original before relying on it.
