Tasks

Parallel Processing

Parallel tasks enable concurrent execution of multiple operations, dramatically improving workflow performance while providing sophisticated control over execution strategies. They're essential for scalable workflows that need to process independent operations simultaneously.

Task Structure

id: parallel-task-example
type: parallel

# Strategy (Required): Determines how task execution and failures are handled
strategy: wait_all  # wait_all | fail_fast | best_effort | race

# Worker Pool (Optional): Control concurrent execution
max_workers: 4      # Maximum concurrent tasks (0 = unlimited)
timeout: 5m         # Overall timeout for all tasks

# Task Definitions (Required): Array of tasks to execute in parallel
tasks:
  - id: task-1
    type: basic
    # ... task configuration

  - id: task-2
    type: basic
    # ... task configuration

# Success Handler (Optional): Execute when parallel task completes
on_success:
  next: downstream-task
  with:
    results: "{{ .output }}"
    execution_time: "{{ .execution.duration }}"

# Error Handler (Optional): Handle failures based on strategy
on_error:
  next: error-handler
  with:
    failed_tasks: "{{ .failed_tasks }}"
    error: "{{ .error }}"

Key Benefits

Performance Multiplier

Transform sequential 30s workflows into 10s parallel execution with smart concurrency control

Strategic Execution

Four execution strategies optimized for different reliability and performance requirements

Resource Intelligence

Adaptive worker management with automatic queuing and timeout handling

Execution Strategies

Strategic control over parallel execution behavior and failure handling:

Default Strategy - Waits for all tasks to complete before proceeding.

Loading diagram...
Comprehensive Data Gathering
id: comprehensive-data-gathering
type: parallel
strategy: wait_all
max_workers: 3
timeout: 10m

tasks:
  - id: task-1
    type: basic
    # ... task configuration

  - id: task-2
    type: basic
    # ... task configuration

Architecture & Execution Flow

Understanding how parallel processing works under the hood:

Loading diagram...

Execution Components

  • Temporal Orchestration

    Distributed workflow engine ensures fault tolerance, automatic retries, and state persistence across system failures

  • Dynamic Worker Allocation

    Intelligent worker pool management with automatic scaling based on max_workers configuration and system resources

  • State Management

    Persistent state tracking for each parallel task with real-time progress monitoring and result aggregation

  • Strategy Engine

    Configurable execution strategies (wait_all, fail_fast, best_effort, race) with intelligent error handling

  • Result Aggregation

    Automatic collection and structuring of outputs accessible via .tasks.<task-id>.output template expressions

Best Practices

Next Steps