Coming from devops background, working on automating usual genomics pipelines, I am benchmarking existing pipelines in order to choose cloud resources properly.
Past week most of work was on testing
I've noticed that
bwa takes about 10 minutes to start processing data. When monitoring
bwa resource usage I've noticed that RAM is slowly building up to about 6GB. That is amount of reference indexed data.
After bwa loads the reference first time, any subsequent run is fast. But why this preloading is happening so slow? My reference data set is already indexed with
When I monitor disk activity with
iotop I noticed that
disk read is only
5MB/s, when I run
bwa mem alignment (only first time), which is very slow. Performance of the drive is lot more than that and a simple
copy test confirms that.
Is it a "limitation" of
bwa, is this slow preloading expected to be like that and I should just accept it and move on forward or there is something I don't understand and maybe someone can help me understand what could I do?
The same "slow preload" happens when I run
bwa shm /path/to/ref.
bwa mem starts immediately. But again slow preload is something that is bothering me and I am trying to understand what can I do about it.