1
$\begingroup$

I have 16 CPUs on my AWS EC2 instance and I am trying to use kSNP4 to generate an SNP matrix. I was able to successfully run this with over 150 samples - while it took a while to run it did complete - and now I want to run it with over 170 samples but when I use the same command it dies at the second step of the process. I have reason to believe it is due to the processors on my server that I am using more than the tool itself.

Does anyone know the best way to ensure maximum processing ability to run tools on an EC2 instance? Or what I can do to increase the power of the instance?

I know this is more than a technical question than a specific bioinformatic question but I would appreciate any insight!

$\endgroup$
3
  • 2
    $\begingroup$ Do you want to increase CPUs or memory? They are very different things. In either case though, this isn't really something we can help with and you should be talking to your sysadmins or AWS help. Sounds like you need a machine with different specs. $\endgroup$
    – terdon
    Commented Feb 12 at 17:32
  • 1
    $\begingroup$ AWS docs outlines how to pick a different instance type for scaling: docs.aws.amazon.com/AWSEC2/latest/UserGuide/… I don't see why a tool that can process 150 samples will crash with 170 unless the complexity goes up geometrically or exponentially somehow $\endgroup$
    – Ram RS
    Commented Feb 12 at 20:36
  • 1
    $\begingroup$ To be honest ... it looks like a bug in the data. 150 -> 170 is not huge and "crashing" is ominous. $\endgroup$
    – M__
    Commented Feb 16 at 17:45

1 Answer 1

1
$\begingroup$

You need to diagnose for low memory and consider switching to an R5 processor that prioritises memory. Basically it's easily solvable by switching hardware. There are certainly alot of other approaches but this is specialist HPC.

Personally, I would use Slurm ParallelCluster in AWS. This is very easy to set up from from the configuration file.

The overview of instance types here. https://aws.amazon.com/ec2/instance-types/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.