Parallel execution

Scheduler options for parallel execution can be provided when executing a workflow

from ewoksdask import execute_graph

result = execute_graph("/path/to/graph.json", scheduler=..., scheduler_options=...)

The different schedulers are

  • multithreading:

    • num_workers: CPU_COUNT by default

  • multiprocessing:

    • context: spawn or fork

    • num_workers: CPU_COUNT by default

  • cluster: scheduler with workers in the current process

    • n_workers: each worker has multiple threads

    • threads_per_worker:

    • processes: subprocesses instead of threads

  • 127.0.0.1:40331: remote scheduler

Remote scheduler

Start a scheduler (+ workers) on any host

from ewoksdask.clusters import local_scheduler
cluster = local_scheduler(n_workers=5)

Separate processes

Start a scheduler on any host

dask scheduler

Add workers to a scheduler with 4 cores each

dask worker 127.0.0.1:8786 --nprocs 4
dask worker 127.0.0.1:8786 --nprocs 4
dask worker 127.0.0.1:8786 --nprocs 4

Slurm scheduler

Start a scheduler on a Slurm submitter host (spawns one Slurm job for each worker)

from ewoksdask.clusters import slurm_scheduler
cluster = slurm_scheduler(maximum_jobs=5)