2
$\begingroup$

For processing genomics workflows I make use of widdle and cromwell with an AWS back-end. I have followed all examples on the AWS cromwell tutorial. In my AWS batch console I can see there are 2 job queues (priority-cromwell and default-cromwell) and both queues can submit jobs to 2 compute environments (ondemand-cromwell and spot-cromwell). I have however no idea how I can control which jobs are put in which queue and are executed in which compute environment. I'd like to make optimal use of the spot-instances, how can I do that? Do I have to add something in the runtime block of my widdle task definitions, should I include something in the cromwell configuration file, or are for example only scattered jobs executed on spot instances?

$\endgroup$
0

1 Answer 1

1
$\begingroup$

Cromwell will send jobs to the AWS Batch queue ARN defined in the cromwell.conf file in the Cromwell server. To use Spot instances you can create an AWS Batch Compute environment configured to use Spot instances and create a queue for that compute environment. You can then set the arn of that queue in the configuration file or override the configured value using the standard mechanisms of overriding Cromwell’s configuration (workflow options, runtime parameters etc). If you change the configuration file you will need to restart the server.

Batch queues can point to more than one compute environment. Each will have a priority. Jobs in a queue will be sent to the highest priority environment until that environment has reached its max vCPU capacity, at that point jobs will be sent to the next environment.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.