Skip to main content

Amazon S3

Nextflow supports Amazon S3. Your pipeline script can access files stored in an S3 bucket transparently, like any other file in the local file system.

S3 paths​

To access an S3 file, prefix the file path with the s3:// scheme and the name of the bucket where it is stored.

For example, if you need to access the file /data/sequences.fa stored in a bucket named my-bucket, that file can be accessed using the following fully qualified path:

s3://my-bucket/data/sequences.fa

The usual file operations can be applied to a path handle with the above notation. For example, the content of an S3 file can be printed as follows:

println file('s3://my-bucket/data/sequences.fa').text

See Working with files and the Path reference to learn more about available file operations.

Authentication​

Nextflow uses the AWS security credentials to make programmatic calls to AWS services.

Nextflow selects AWS credentials from the following sources, in descending order of priority:

  1. Nextflow configuration file - aws.accessKey and aws.secretKey. See aws for more details.

  2. A custom profile in $HOME/.aws/credentials, $HOME/.aws/config, or both. Set the profile with the aws.profile config option or the AWS_PROFILE or AWS_DEFAULT_PROFILE environment variables.

  3. Environment variables - AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY, and optionally AWS_SESSION_TOKEN for temporary credentials.

  4. Web identity token credentials, such as IAM roles for service accounts in Amazon EKS. Configure these with the AWS_WEB_IDENTITY_TOKEN_FILE and AWS_ROLE_ARN environment variables.

  5. The default profile in ~/.aws/credentials, ~/.aws/config, or both.

  6. New in 23.10: Single Sign-On (SSO) credentials. See the AWS documentation for more details.

  7. Container credentials, such as the task IAM role in Amazon ECS.

  8. EC2 instance profile credentials. See the AWS documentation and this blog post for more details.

Nextflow selects the AWS region from the following sources, in descending order of priority:

  1. Nextflow configuration file: aws.region
  2. Environment variables: AWS_REGION or AWS_DEFAULT_REGION
  3. EC2 instance metadata (if Nextflow is running in an EC2 instance)

SSO credentials and instance profile credentials are recommended because you do not need to manage and distribute AWS keys. Use SSO credentials to launch pipelines from outside AWS, for example, from your laptop. Use instance profile credentials to launch pipelines within AWS, for example, from an EC2 instance.

If Nextflow finds no credentials, it accesses S3 anonymously. Anonymous access works only for public buckets.

Required permissions​

AWS uses IAM policies to define permissions for IAM identities. Attach the required policies to the identity associated with your AWS credentials.

Nextflow requires policies that grant access to S3 buckets to use the work directory, pull input data, and publish results.

Depending on the pipeline configuration, these actions can use a single bucket but more often span multiple buckets. After you identify the buckets your pipeline uses, give Nextflow access to them in one of two ways:

  1. Grant access to all buckets by attaching the policy "s3:*" to the IAM identity. This works only if buckets do not set their own access policies (see option 2).

  2. For more fine-grained control, assign the following policy to each bucket, replacing the placeholders with actual values:

    {
        "Version": "2012-10-17",
        "Id": "<policy-id>",
        "Statement": [
            {
                "Sid": "<statement-id>",
                "Effect": "Allow",
                "Principal": {
                    "AWS": "<nextflow-identity-arn>"
                },
                "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:DeleteObject",
                    "s3:PutObjectTagging",
                    "s3:AbortMultipartUpload"
                ],
                "Resource": "arn:aws:s3:::<bucket-name>/*"
            },
            {
                "Sid": "AllowSSLRequestsOnly",
                "Effect": "Deny",
                "Principal": "*",
                "Action": "s3:*",
                "Resource": [
                    "arn:aws:s3:::<bucket-name>",
                    "arn:aws:s3:::<bucket-name>/*"
                ],
                "Condition": {
                    "Bool": {
                        "aws:SecureTransport": "false"
                    }
                }
            }
        ]
    }

See the bucket policy documentation for more details.

See AWS Batch for the additional permissions required by the AWS Batch executor.

Custom endpoints​

Use the aws.client.endpoint config option to specify a custom S3 endpoint. For example, to use an AWS China region, specify the corresponding S3 endpoint:

aws {
    client {
        endpoint = 'https://s3.cn-north-1.amazonaws.com.cn'
    }
}

See the AWS documentation for the list of S3 endpoints.

S3-compatible storage​

To use S3-compatible object storage such as Ceph or Minio, specify the endpoint of your storage provider and enable S3 path style access in your Nextflow configuration:

aws {
    accessKey = '<aws-access-key>'
    secretKey = '<aws-secret-key>'
    client {
        endpoint = '<storage-endpoint-url>'
        s3PathStyleAccess = true
    }
}

Advanced configuration​

See the aws config scope for all available options.

On this Page