Skip to main content

Executors

In Nextflow, the executor is the component that runs the tasks submitted by a pipeline.

The executor provides an abstraction between the pipeline definition (what it computes) and the underlying execution system (how it runs). You can write a pipeline once and run it on your local machine, a high-performance computing (HPC) cluster, or in the cloud by changing the executor in the Nextflow configuration.

Executors share the concepts and usage patterns described below. For the complete list of executors and their supported directives, see Executor reference.

Selecting an executor​

By default, Nextflow runs tasks with the local executor, on the machine where Nextflow is launched. To use a different executor, set process.executor:

process.executor = 'slurm'

The executor is a process directive. Use process selectors to run different processes with different executors.

How tasks are executed​

For each task, Nextflow creates a task directory in the work directory and writes the task script (.command.sh) and a wrapper script (.command.run). The wrapper script stages the input files, runs the task script, and saves the outputs and exit status. The executor submits the wrapper script to the underlying execution system. Nextflow then polls the status of each submitted task at a regular interval (see executor.pollInterval) until it completes.

Types of executors​

Executors fall into the following categories:

  • Local: The local executor runs tasks as processes on the machine where you launch Nextflow. Use it for development and for small pipelines.

  • Cloud: Cloud executors submit each task to a managed compute service or container platform. Every process must specify a container.

  • HPC schedulers: Grid executors such as SLURM, LSF, and SGE submit each task as a job to a cluster resource manager. Launch the pipeline from a node that can submit jobs, typically the cluster login node. Use a work directory on a shared file system that all compute nodes can access, or in object storage when using Fusion.

Configuring executors​

Use the executor config scope to control executor behavior, such as how many tasks can be queued in parallel and how often to poll for task status:

executor {
    queueSize = 100
    pollInterval = '30sec'
}

To apply executor config settings to a specific executor, prefix the executor name with $ and use it as a special scope. For example:

executor {
    $sge {
        queueSize = 100
        pollInterval = '30sec'
    }

    $local {
        cpus = 8
        memory = '32 GB'
    }
}

Settings without an executor name prefix apply to all executors. Some settings have different default values for each executor. See executor for the list of available settings and their defaults.

Many executors also have a dedicated config scope for platform-specific settings, such as aws, azure, google, and k8s.

Hybrid workloads​

Nextflow can use multiple executors in the same pipeline. Use this to run hybrid workloads, in which some tasks run locally or on an HPC cluster and other tasks run in the cloud.

Use process selectors to apply a cloud executor to the processes that you want to offload. For example:

process {
    withLabel: bigTask {
        executor = 'awsbatch'
        queue = 'my-batch-queue'
        container = 'my/image:tag'
    }
}

aws.region = 'eu-west-1'

With this configuration, processes with the bigTask label run on AWS Batch, while the remaining processes run with the local executor.

Use the -bucket-dir option (or the bucketDir config option) to specify a remote work directory for the cloud tasks and, optionally, -work-dir to specify a local work directory for the local tasks:

nextflow run <pipeline> -bucket-dir s3://<bucket>/work

Nextflow automatically transfers input and output files between the local and remote work directories.

warning

The remote work directory must contain at least one subdirectory (for example, s3://<bucket>/work rather than s3://<bucket>).

note

When using Fusion, all tasks use the remote work directory. Specify workDir instead of bucketDir.

On this Page