Pattern for Parallel Sub-Workflow Execution Followed by Wait-For-All Loop

What this workflow does

This (set of) workflow(s) shows how to start multiple sub-workflows, asynchronously, in parallel, and then wait for all of them to complete. Normally sub-workflows would need to be run synchronously, in series, or, if they are executed asynchronously (to run concurrently, in parallel), there is no easy way to merge/wait for an arbitrary number of them to complete.

This is a "design pattern" template to show one approach for running multiple, data-driven instances of a sub-workflow "asynchronously," in parallel (instead of running them one at a time in series), but still prevent the later steps in the workflow from continuing until all of the sub-workflows have reported back that they are finished, via callback URL. There are other techniques involving messaging services, database tables, or other external "flow manager" helpers, but this technique accomplishes the goal fully within n8n.

Setup

To implement this pattern, examine the nodes in the template and modify the incoming data leading to:

1. A split-out loop to acynchronously execute a sub-workflow multiple times, in parallel.
   • For instance, each sub-workflow might process one of a list of incoming documents.
   • The resumeUrl for the main/parent workflow is provided to all of the sub-workflow executions, along with a unique identifier that can be counted later (e.g. a document file-name).
2. A "wait-for-all" loop that checks whether all sub-workflows have reported back (if node) and builds a unique list of identifiers from the callbacks received from each execution of the sub-workflow.
   • The sub-workflow should be designed to respond immediately (async) and later send a callback request when it has finished processing.
   • The callback request should include the unique identifier value received when the sub-workflow it was started.

This is meant to be a possible answer to questions like this one about running things in parallel, maybe this one about waiting for things to finish, this one about managing sub-batches of things by waiting for each batch, or this one about running things in parallel. The topic of how to do this comes up A LOT, and this is one of the only techniques that (so far) seems to work.