Coming from devops background, working on automating usual genomics pipelines, I am benchmarking existing pipelines in order to choose cloud resources properly.

Past week most of work was on testing bwa mem.

I've noticed that bwa takes about 10 minutes to start processing data. When monitoring bwa resource usage I've noticed that RAM is slowly building up to about 6GB. That is amount of reference indexed data.

After bwa loads the reference first time, any subsequent run is fast. But why this preloading is happening so slow? My reference data set is already indexed with bwa index.

When I monitor disk activity with iotop I noticed that disk read is only 5MB/s, when I run bwa mem alignment (only first time), which is very slow. Performance of the drive is lot more than that and a simple copy test confirms that.

Is it a "limitation" of bwa, is this slow preloading expected to be like that and I should just accept it and move on forward or there is something I don't understand and maybe someone can help me understand what could I do?

The same "slow preload" happens when I run bwa shm /path/to/ref. Then bwa mem starts immediately. But again slow preload is something that is bothering me and I am trying to understand what can I do about it.

• Issue was with EBS snapshot which holds reference indexed data. Issue is that when you run EC2 from snapshot, EBS storage performance is expected to be low until you optimize the volume. My assumptions and questions for two weeks, all "goes to the water". Unfortunately I was not aware of this hence I asked the question out from ignorance. Aug 8 at 22:14
• Can you please add this as an actual answer and accept it? This will help the StackExchange algorithms to identify this as a question with and answer, and stop promoting it in the future to fish for more answers.
– gringer
Aug 8 at 23:22
• I will. Didn not want to do that imediatelly since it is a little bit awkward to answer own question = ) Aug 9 at 0:15
• Yeah, I find it awkward as well, but you did already answer the question (as a comment to your own question). Might as well get a bit more credit for that. Also, it's worth thinking about comments as transient things that are there only to help people improve their work; they're not meant to be used for anything substantial that should hang around for a long time (beyond gratitude).
– gringer
Aug 9 at 2:52

I'll add an answer to my own question, which was asked out of lack of experience with cloud (AWS) EC2 service.

The way I was testing bwa is from a custom AMI image created for our pipelines.

The image has "preloaded" reference data that we are using for alignement so that we spare some time downloading reference data from S3 and spare some S3 api calls since we have a lot of files used as reference data.

Simply our image is single 200GB EBS volume used with AWS ECS AMI where we stored our reference data and created a custom image for Batch out of it. Then we use that AMI for AWS Batch jobs.

Once I used iotop (after few days of testing) to monitor disk activity I've noticed that bwa loads index into memory very slow, cca 5MB/s. Syncing files from S3 was faster then using this snapshot approach.

For benchmarking I've used the same AMI manually launched. In the AWS documentation there is part which explains why and how you should optimize EBS when AMI is launched from a snapshot. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

After spining up fresh EC2, syncing reference data and running bwa everything was clear. My EBS volume was slow and had to be initialized first.

That is all. My lack of EBS storage knowledge was the issue.

This helped me understand that I should not use custom AMI with ref data stored and will probably use EFS or simply syncing the data from S3 inside Docker entrypoint.

Answer from @user438383, converted from comment:

BWA has slow indexing due to the indexing size; this can be improved by using bwa-mem2:

https://github.com/bwa-mem2/bwa-mem2

We are happy to announce that the index size on disk is down by 8 times and in memory by 4 times due to moving to only one type of FM-index (2bit.64 instead of 2bit.64 and 8bit.32) and 8x compression of suffix array. For example, for human genome, index size on disk is down to ~10GB from ~80GB and memory footprint is down to ~10GB from ~40GB. There is a substantial reduction in index IO time due to the reduction and hardly any performance impact on read mapping.

• Yes but it is not yet production ready. Talking from my very poor amount of reading on the bwa-mem2 topics. Aug 9 at 0:57