2
$\begingroup$

I am in the process of creating a bioinformatics pipeline and I am currently trying to setup the AWS infrastructure for which it will run.

The script I am running to setup the AWS batch job looks like the following:

#! /bin/bash

# Setup the AWS Batch architecture
# Copyright 2022, SolareaBio

set -e


# ***** BEGIN custom lines

container="BGC_bacteria"
vCPUs_req=12
memory_MiB_req=40000
# ASK ABOUT CUSTOM CPU AND MEMORY SETTINGS

# ***** END custom lines



container_lc=$(echo $container | tr [A-Z] [a-z])

subnet_useast1a="subnet-42371d0a"
subnet_useast1b="subnet-b03c54ea"
subnet_useast1c="subnet-00ee1364"

usage() {
  cat << EOF
Usage $0 [-h]

Builds the $container docker container, pushes it to the AWS ECR, and sets up the $container AWS Batch environment
EOF
}

while getopts "h" option; do
  case $option in
    h)   usage;exit;;
    [?]) usage; exit;;
  esac
done

script_dir=$(dirname "$0")

repo="990850896279.dkr.ecr.us-east-1.amazonaws.com"

echo "Logging into ECR"
aws ecr get-login-password --region us-east-1 | docker login --username AWS --password-stdin ${repo}

image_and_tag="${container_lc}:latest"
echo "Building the docker container: $image_and_tag"

# If you get errors from the docker build command, try adding --no-cache
docker build --no-cache -f $script_dir/${container}.dockerfile -t $image_and_tag $script_dir

echo "Tagging Image"
docker tag ${image_and_tag} ${repo}/${image_and_tag}

# Create repository in AWS ECR
repo_container_info=$(aws ecr describe-repositories --repository-names ${container_lc})
if [[ "$repo_container_info" == "" ]]; then
  aws ecr create-repository --repository-name ${container_lc}
fi

echo "Pushing Image up to AWS"
docker push ${repo}/${image_and_tag}

# Create compute environment
EC2_type=ON_DEMAND
ce_info=$(aws batch describe-compute-environments --compute-environments ${container}_${EC2_type}_CE)
ce_status=$(echo $ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
if [[ $ce_status != "VALID" ]]; then
  compute_resources='{"type":"EC2","allocationStrategy":"BEST_FIT_PROGRESSIVE","minvCpus":0,"maxvCpus":10000,"tags":{"Name":"'${container}_Batch'"},"instanceTypes":["c5a","m5a","c5","m5"],"subnets":["'$subnet_useast1a'","'$subnet_useast1b'","'$subnet_useast1c'"],"securityGroupIds":["sg-ff2fd48f","sg-65ff5315"],"ec2KeyPair":"SBTools","instanceRole":"ecsInstanceRole"}'
  aws batch create-compute-environment \
    --compute-environment-name ${container}_${EC2_type}_CE \
    --type MANAGED \
    --state ENABLED \
    --compute-resources $compute_resources \
    --service-role AWSBatchServiceRole

  printf "Waiting for Compute Environment to be created"
  while [[ $ce_status != "VALID" ]]; do
sleep 5
printf "."
ce_info=$(aws batch describe-compute-environments --compute-environments ${container}_${EC2_type}_CE)
    ce_status=$(echo $ce_info | jq '.computeEnvironments[0] | .status // ""' | tr -d '"')
  done
  echo ""
fi

# Create the job queue
jq_info=$(aws batch describe-job-queues --job-queues ${container}_${EC2_type}_JQ)
jq_status=$(echo $jq_info | jq '.jobQueues[0] | .status // ""' | tr -d '"')
if [[ $jq_status != "VALID" ]]; then
  aws batch create-job-queue \
    --job-queue-name ${container}_${EC2_type}_JQ \
    --state ENABLED \
    --priority 5 \
    --compute-environment-order '{"order":1,"computeEnvironment":"'${container}_${EC2_type}_CE'"}'
fi

# Register job definition
job_definition="${container}_JD"
jd_info=$(aws batch describe-job-definitions --status ACTIVE --job-definition-name $job_definition)
jd_status=$(echo $jd_info | jq '.jobDefinitions[0] | .status // ""' | tr -d '"')
if [[ $jd_status != "ACTIVE" ]]; then
  aws batch register-job-definition \
    --job-definition-name $job_definition \
    --type container \
    --container-properties '{"privileged":true,"image":"'$repo/$image_and_tag'","vcpus":'${vCPUs_req}',"memory":'${memory_MiB_req}',"mountPoints":[{"sourceVolume":"SB-EFS","containerPath":"/EFS"}],"volumes":[{"host":{"sourcePath":"/"},"name":"SB-EFS"}],"command":["Ref::container","Ref::run_id","Ref::input_file"]}' \
    --retry-strategy '{"attempts": 10,"evaluateOnExit":[{"onStatusReason" :"Host EC2*","action": "RETRY"},{"onReason" : "*","action": "EXIT"}]}'
fi

# Make sure the EFS has an access point set up
efs_id=$(aws efs describe-file-systems | jq -r '.FileSystems[] | select (.Name=="SB-EFS") | .FileSystemId')
if [[ "$efs_id" == "" ]]; then
  echo "ERROR: SB-EFS does not exist"
  exit
fi
ap_id=$(aws efs describe-access-points | jq -r '.AccessPoints[] | select (.Name=="EFS_AP") | .AccessPointId')
if [[ "$ap_id" == "" ]]; then
  # 1001:1001 is user sb in group sb
  ap_id=$(aws efs create-access-point --file-system-id "$efs_id" --posix-user '{"Uid":1001,"Gid":1001}' --root-directory '{"Path":"/"}' --tags '[{"Key":"Name","Value":"EFS_AP"}]' | jq -r '.AccessPointId')
fi  

Now when I run it, everything works as it should until getting to the --retry-strategy command line in the "register job definition" section of the script. The following error is subsequently thrown:

Parameter validation failed:
Unknown parameter in retryStrategy: "evaluateOnExit", must be one of: attempts

Not sure what the issue is with the "attempts" parameter that is set. Any potential solutions or guidance to solving this error is much appreciated. Thanks!

$\endgroup$
2
  • 1
    $\begingroup$ Hi @Joe_G, could you please provide a bit more information about what you're trying to do with this script? Stories help a lot with getting good, helpful answers. I can see something suggesting metagenomics, but it's not clear to me what a correct script would do. $\endgroup$
    – gringer
    Oct 28, 2022 at 21:08
  • 1
    $\begingroup$ My personal approach would be to use the NextFlow AWS pipeline. It would simplify everything, e.g. NextFlow automatically 'Dockerises' $\endgroup$
    – M__
    Oct 30, 2022 at 0:00

1 Answer 1

0
$\begingroup$

You didn't say what version of the AWS CLI you're using, but I suspect it's version 1. It looks like the evaluateOnExit feature was added in version 2 but was not backported. Installing AWS CLI version 2 resolves the issue:

https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html

Note that Nextflow also provides built-in support for AWS Batch. You'd still need to configure some AWS infrastructure using Nextflow, but it should be a lot less painful than writing/maintaining your own script(s). It's definitely worth checking out if you haven't already.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.