Overview

The rise of big data has made it increasingly necessary to be able to analyze and perform experiments on large datasets in a portable and reproducible manner. Parallelization and distributed computing are the best ways to tackle this challenge. However, the tools commonly available to computational scientists often lack good support for these techniques or provide a model that fits poorly with the needs of computational scientists and require knowledge of complex tools and APIs. Nextflow was created to address these challenges.

The Nextflow language is inspired by the Unix philosophy, in which many simple command line tools can be composed into powerful workflows. Similarly, a Nextflow script consists of many simple processes composed into pipelines. Each process executes a given tool or script in any supported language, and by specifying the process inputs and outputs, Nextflow coordinates the execution of tasks for you.

The Nextflow runtime integrates with popular execution platforms (i.e., HPC schedulers and cloud providers) and software tools (e.g., Git, Docker, and Conda), allowing you to fully describe a computational pipeline with all of its dependencies and run it in nearly any environment - write once, run anywhere.

Processes and dataflow

A Nextflow pipeline consists of processes connected by dataflow logic. You can write each process in any scripting language, such as Bash, Perl, Python, or R.

Each process has inputs and outputs. Data flows from process to process through dataflow channels and dataflow values. The data dependencies between these processes implicitly determine the flow of execution. For example:

// Enable static typing (Nextflow 26.10 or later)
nextflow.enable.types = true

// Pipeline parameters
params {
    query: Path = '/some/data/sample.fa'
    db: Path = '/some/path/pdb'
}

// Define the workflow
workflow {
    query_ch = channel.of(params.query)
    hits_ch = blast_search(query_ch, params.db)
    extract_top_hits(hits_ch, params.db).view()
}

// Define the blast_search process
process blast_search {
    input:
    query: Path
    db: Path

    output:
    file('top_hits.txt')

    script:
    """
    blastp -db $db -query $query -outfmt 6 > blast_result
    cat blast_result | head -n 10 | cut -f 2 > top_hits.txt
    """
}

// Define the extract_top_hits process
process extract_top_hits {
    input:
    top_hits: Path
    db: Path

    output:
    file('sequences.txt')

    script:
    """
    blastdbcmd -db $db -entry_batch $top_hits > sequences.txt
    """
}

The above example defines two processes. Their execution order is determined by their dependencies, not the order in which they are called. Because extract_top_hits depends on the output of blast_search, blast_search runs first, then extract_top_hits.

When the workflow runs, it creates two processes (blast_search and extract_top_hits) connected by the hits_ch channel. Each process executes a task and emits a value for each input that it receives. Whenever blast_search emits a value, extract_top_hits receives it through the hits_ch channel.

See Dataflow to learn more.

Execution abstraction

Nextflow abstracts the pipeline logic from the underlying execution environment. While a process defines what task is executed, the executor determines how that task is executed in the target environment. You write a pipeline once and run it on your computer, an HPC cluster, or the cloud by selecting a different executor in your configuration.

By default, Nextflow executes tasks on the local computer. The local executor is useful for development and testing, but real-world computational pipelines usually require an HPC or cloud environment. Nextflow supports many HPC and cloud executors, including SLURM, IBM Spectrum LSF, AWS Batch, and Google Cloud Batch.

See Executors to learn more.

Scripting language

Nextflow is a workflow language for writing scalable and reproducible pipelines. It is based on Java and Groovy, but its syntax is familiar if you know other scripting languages such as Python and R.

See Scripts to learn more.

Configuration options

You define pipeline configuration settings in a nextflow.config file in the launch directory. The configuration file specifies how the pipeline is executed, including settings such as the executor and other process directives. For example:

process {
    executor = 'sge'
    queue = 'cn-el6'
}

See Configuration to learn more.