Skip to main content

Custom AMIs for AWS Batch

Create a custom Amazon Machine Image (AMI) for the compute environments that the AWS Batch executor uses.

tip

The main reason to use a custom AMI is to provide the AWS CLI to task containers. This is a legacy requirement. Consider using Wave and Fusion to access S3 without the AWS CLI.

You might need to create your own AMI for your compute environments when:

  • You want to provide the AWS CLI through the AMI instead of installing it in each Docker image
  • The existing AMI (selected from the marketplace) does not have Docker installed
  • You need to attach more storage to your EC2 instance (the default ECS instance AMI has only a 30 GB EBS volume, which is not enough for most data pipelines)
  • You need to install additional software that is not available in your Docker image

Create your custom AMI​

From the EC2 Dashboard, select Launch Instance, then select Browse more AMIs. On the new page, select AWS Marketplace AMIs, and then search for Amazon ECS-Optimized Amazon Linux 2023 (AL2023) x86_64 AMI. Select the AMI and continue as usual to configure and launch the instance.

note

The selected instance has a root volume of 30 GB. Increase its size or add a second EBS volume with enough storage for genomic workloads.

When the instance is running, SSH into it (or connect with the Session Manager service), install the AWS CLI, and install any other tools you need (see the following sections).

Finally, select Create Image from the EC2 Dashboard to create a new AMI from the running instance. You can also use the AWS CLI.

Specify the new AMI ID when you create the Batch compute environment.

warning

Any additional software must be installed on the EC2 instance before creating the AMI.

AWS CLI installation​

Install the AWS CLI in your custom AMI with a self-contained package manager such as Conda. With Conda, you control which Python version the AWS CLI uses (the AWS CLI is written in Python).

Without Conda, the aws command tries to use the Python version installed in the container and can't find the necessary dependencies.

The following snippet installs the AWS CLI with Miniconda in the home folder:

cd $HOME
sudo yum install -y bzip2 wget
wget https://repo.continuum.io/miniconda/Miniconda3-latest-Linux-x86_64.sh
bash Miniconda3-latest-Linux-x86_64.sh -b -f -p $HOME/miniconda
$HOME/miniconda/bin/conda install -c conda-forge --override-channels -y awscli
rm Miniconda3-latest-Linux-x86_64.sh

Then verify that the AWS CLI works:

$ ./miniconda/bin/aws --version
aws-cli/1.29.20 Python/3.11.4 Linux/4.14.318-241.531.amzn2.x86_64 botocore/1.31.20
note

The aws tool is installed in a directory named bin in the main installation folder. Modifying this directory structure after installation breaks the tool.

Then set aws.batch.cliPath to the path of the aws tool in your AMI (for example, /home/ec2-user/miniconda/bin/aws).

Docker installation​

Nextflow requires Docker to execute tasks on AWS Batch. The Amazon ECS-optimized Amazon Linux 2023 AMI includes Docker. If you create your AMI from a different AMI without Docker, install Docker manually.

The following snippet installs Docker on an Amazon EC2 instance:

# install Docker
sudo yum update -y
sudo yum install docker

# start the Docker service
sudo service docker start

# empower your user to run Docker without sudo
sudo usermod -a -G docker ec2-user

Log out and log back in to use the new ec2-user permissions.

Complete these steps before you create the AMI from the current EC2 instance.

Amazon ECS container agent installation​

The ECS container agent is a component of Amazon Elastic Container Service (Amazon ECS) that manages containers on behalf of ECS. AWS Batch uses ECS to execute containerized jobs. It requires the agent on the EC2 instances in your compute environments.

The ECS agent is included in the Amazon ECS-optimized Amazon Linux 2023 AMI. If you use a different base AMI, you can install the agent on any EC2 instance that supports the Amazon ECS specification.

To install the agent:

sudo yum install ecs-init
sudo systemctl enable --now ecs

To test the installation:

curl -s http://localhost:51678/v1/metadata | python -mjson.tool
note

Attach the AmazonEC2ContainerServiceforEC2Role policy to the instance role so that the EC2 instance created by the compute environment can register with the ECS cluster.

note

Attach the AmazonEC2ContainerRegistryReadOnly policy to the instance role to get read-only access to Amazon Elastic Container Registry (ECR) repositories.

On this Page