2
$\begingroup$

My current job is rate-limited chiefly by the IO overhead of writing reads to their corresponding bam files (they are split up by both cell barcode and contig). The only reason I need to write these mini-split bams is so that I can then use the pysam coverage function to assess read depth at certain locations. I already have all of these reads in memory as a list of AlignedSegments. Is it necessary to write them to a .bam file in order to be able to index and use the coverage function? I feel like there should be a way to have the AlignedSegments be treated as a bam object in memory, possibly even indexed in memory, and then treated as if they were just streamed from a bamfile with an index in order to be able to use the coverage method. Thanks in advance, if this is possible it would increase my processing speed x10. Or, given a list of AlignedSegments, is there a way to get coverage at specific positions across all those AlignedSegments in my list (that would get me what I need).

$\endgroup$
3
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – Community Bot
    Oct 14, 2023 at 2:05
  • $\begingroup$ What operating system are you using? $\endgroup$
    – terdon
    Oct 14, 2023 at 13:21
  • $\begingroup$ My operating system is linux $\endgroup$
    – ekofman
    Oct 16, 2023 at 0:36

1 Answer 1

3
$\begingroup$

Assuming you are on an operating system that supports them, like Linux, you can try using named pipes in a ramdisk or tmpfs. Not the most elegant approach, but I don't know if what you are asking for is possible (it may very well be, I just don't know) and this can be a decent workaround.

First, check that you do have a mounted tmpfs. On my system, I have a few:

$ mount | awk '$1=="tmpfs"'
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,inode64)
tmpfs on /tmp type tmpfs (rw,nosuid,nodev,size=16278892k,nr_inodes=1048576,inode64)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=3255776k,nr_inodes=813944,mode=700,uid=1000,gid=1000,inode64)

These filesystems are actually stored in RAM and should be significantly faster. For instance, on my machine, which has an SSD as the main storage, I still get significant speedup when using /dev/shm:

## SSD
$ time seq 1000000000 > file
real    0m13.687s
user    0m5.623s
sys     0m7.516s

$ time wc -l < file
1000000000

real    0m4.972s
user    0m0.101s
sys     0m2.248s

## tmpfs, RAM
$ time seq 1000000000 > /dev/shm/file

real    0m6.755s
user    0m4.982s
sys     0m1.763s

$ time wc -l < /dev/shm/file
1000000000

real    0m1.024s
user    0m0.120s
sys     0m0.902s

As you can see, using /dev/shm is significantly faster. And using named pipes can give you a little boost too:

$ time ( seq 1000000000 > /dev/shm/file;  wc -l < /dev/shm/file)
1000000000

real    0m8.341s
user    0m5.277s
sys     0m3.044s

$ rm /dev/shm/file && mkfifo /dev/shm/file 
$ time ( seq 1000000000 > /dev/shm/file &  wc -l < /dev/shm/file)
1000000000

real    0m6.564s
user    0m5.426s
sys     0m3.965s

So, this is essentially a way of doing everything in RAM without needing to code it as doing everything in RAM. You can create these files (the FIFOs, the named pipes) and treat them as files, but since you're creating them in a tmpfs filesystem, they're actually in RAM and by using FIFOs you not only get a further speed jump but also minimise the actual data that need to be stored.

$\endgroup$
1
  • $\begingroup$ This seems like it gets at what I'm trying to accomplish. I will try it! Thank you. $\endgroup$
    – ekofman
    Oct 16, 2023 at 0:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.