Parallel Processing
Parallel tasks enable concurrent execution of multiple operations, dramatically improving workflow performance while providing sophisticated control over execution strategies. They're essential for scalable workflows that need to process independent operations simultaneously.
Task Structure
id: parallel-task-example
type: parallel
# Strategy (Required): Determines how task execution and failures are handled
strategy: wait_all # wait_all | fail_fast | best_effort | race
# Worker Pool (Optional): Control concurrent execution
max_workers: 4 # Maximum concurrent tasks (0 = unlimited)
timeout: 5m # Overall timeout for all tasks
# Task Definitions (Required): Array of tasks to execute in parallel
tasks:
- id: task-1
type: basic
# ... task configuration
- id: task-2
type: basic
# ... task configuration
# Success Handler (Optional): Execute when parallel task completes
on_success:
next: downstream-task
with:
results: "{{ .output }}"
execution_time: "{{ .execution.duration }}"
# Error Handler (Optional): Handle failures based on strategy
on_error:
next: error-handler
with:
failed_tasks: "{{ .failed_tasks }}"
error: "{{ .error }}"
Key Benefits
Performance Multiplier
Strategic Execution
Resource Intelligence
Execution Strategies
Strategic control over parallel execution behavior and failure handling:
Default Strategy - Waits for all tasks to complete before proceeding.
id: comprehensive-data-gathering
type: parallel
strategy: wait_all
max_workers: 3
timeout: 10m
tasks:
- id: task-1
type: basic
# ... task configuration
- id: task-2
type: basic
# ... task configuration
Architecture & Execution Flow
Understanding how parallel processing works under the hood:
Execution Components
- Temporal Orchestration
Distributed workflow engine ensures fault tolerance, automatic retries, and state persistence across system failures
- Dynamic Worker Allocation
Intelligent worker pool management with automatic scaling based on max_workers configuration and system resources
- State Management
Persistent state tracking for each parallel task with real-time progress monitoring and result aggregation
- Strategy Engine
Configurable execution strategies (wait_all, fail_fast, best_effort, race) with intelligent error handling
- Result Aggregation
Automatic collection and structuring of outputs accessible via
.tasks.<task-id>.output
template expressions