Slow bwa shm "preloading" into memory

Question

Coming from devops background, working on automating usual genomics pipelines, I am benchmarking existing pipelines in order to choose cloud resources properly.

Past week most of work was on testing bwa mem.

I've noticed that bwa takes about 10 minutes to start processing data. When monitoring bwa resource usage I've noticed that RAM is slowly building up to about 6GB. That is amount of reference indexed data.

After bwa loads the reference first time, any subsequent run is fast. But why this preloading is happening so slow? My reference data set is already indexed with bwa index.

When I monitor disk activity with iotop I noticed that disk read is only 5MB/s, when I run bwa mem alignment (only first time), which is very slow. Performance of the drive is lot more than that and a simple copy test confirms that.

Is it a "limitation" of bwa, is this slow preloading expected to be like that and I should just accept it and move on forward or there is something I don't understand and maybe someone can help me understand what could I do?

The same "slow preload" happens when I run bwa shm /path/to/ref. Then bwa mem starts immediately. But again slow preload is something that is bothering me and I am trying to understand what can I do about it.

Answer 1

I'll add an answer to my own question, which was asked out of lack of experience with cloud (AWS) EC2 service.

The way I was testing bwa is from a custom AMI image created for our pipelines.

The image has "preloaded" reference data that we are using for alignement so that we spare some time downloading reference data from S3 and spare some S3 api calls since we have a lot of files used as reference data.

Simply our image is single 200GB EBS volume used with AWS ECS AMI where we stored our reference data and created a custom image for Batch out of it. Then we use that AMI for AWS Batch jobs.

Once I used iotop (after few days of testing) to monitor disk activity I've noticed that bwa loads index into memory very slow, cca 5MB/s. Syncing files from S3 was faster then using this snapshot approach.

For benchmarking I've used the same AMI manually launched. In the AWS documentation there is part which explains why and how you should optimize EBS when AMI is launched from a snapshot. https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-initialize.html

After spining up fresh EC2, syncing reference data and running bwa everything was clear. My EBS volume was slow and had to be initialized first.

That is all. My lack of EBS storage knowledge was the issue.

This helped me understand that I should not use custom AMI with ref data stored and will probably use EFS or simply syncing the data from S3 inside Docker entrypoint.

Answer 2

Answer from @user438383, converted from comment:

BWA has slow indexing due to the indexing size; this can be improved by using bwa-mem2:

https://github.com/bwa-mem2/bwa-mem2

We are happy to announce that the index size on disk is down by 8 times and in memory by 4 times due to moving to only one type of FM-index (2bit.64 instead of 2bit.64 and 8bit.32) and 8x compression of suffix array. For example, for human genome, index size on disk is down to ~10GB from ~80GB and memory footprint is down to ~10GB from ~40GB. There is a substantial reduction in index IO time due to the reduction and hardly any performance impact on read mapping.