Timeline for How to get strain names/ids contained in a multi FASTA file using seqkit?
Current License: CC BY-SA 4.0
21 events
| when toggle format | what | by | license | comment | |
|---|---|---|---|---|---|
| May 31, 2022 at 22:41 | comment | added | M__♦ |
Thanks @terdon its surely grep/ripgrep. If seqkit is chunking at source, that is really interesting (its clearly not pulling the whole file in and then chunking).
|
|
| May 31, 2022 at 22:08 | comment | added | M__♦ | @CorneliusRoemer if it is chunking it at source, its not a bug. It would make sense. Does it parallelise? Chunking and parallelising would be a technique to speed the operation and minimise RAM issues. Obviously it will fall over via a network. | |
| May 31, 2022 at 19:04 | history | edited | terdon♦ | CC BY-SA 4.0 |
deleted 5 characters in body
|
| May 31, 2022 at 18:59 | history | edited | terdon♦ | CC BY-SA 4.0 |
added 168 characters in body
|
| May 31, 2022 at 18:55 | comment | added | terdon♦ |
@M__ my initial numbers don't hold up when doing the same thing with a fasta file stored on a local SSD. So it looks like seqkit is slower than expected when reading data over a network but is faster than everything except the simple grep + tr when running on a local file.
|
|
| May 31, 2022 at 18:54 | comment | added | terdon♦ |
@CorneliusRoemer Ha! Serves me right for throwing my age around :) I do indeed have 20 years of experience, but I still come from the world of biology, not programming so there will be holes in my knowledge. I have updated the post with the new results, including all data needed to reproduce them, and seqkit now only loses out to grep + tr but is faster than all the others. So much for my years of experience, I have to eat my words now!
|
|
| May 31, 2022 at 18:52 | history | edited | terdon♦ | CC BY-SA 4.0 |
added 1892 characters in body
|
| May 31, 2022 at 18:50 | comment | added | Cornelius Roemer |
Ah interesting! I thought you were the bioinformatician with 20y of experience. I barely have 1 :D Seqkit could read in chunks? No idea, one should maybe even open this as a bug on Github. By the way there are other similar tools out there. Please try rg too, it's a Rust port of grep and likely faster than grep. Also best to link to the data file so it's reproducible. Performance could differ on macOS and *nix. Ah yes, that's an advantage of seqkit it could possibly work even on windows...
|
|
| May 31, 2022 at 18:34 | comment | added | terdon♦ |
@CorneliusRoemer good catch! The test results I show were using a genome file stored on a network drive, which would of course affect the times. I had assumed that wouldn't be relevant since all tools were accessing the same file over the same network. However, I downloaded a different genome file locally and on my local SSD, seqkit is far more competitive. I'll update in a few minutes with the new test results, but do you have any idea why one tool would be affected by the network I/O times more than another? That's very surprising to me.
|
|
| May 31, 2022 at 18:10 | comment | added | Cornelius Roemer |
That's a good reason to use an alternative tool, I agree, if speed matters. If you use the command line interactively though, terseness can be more important. It depends on the use case I suppose. I'd be curious how fast rg would be. The slowdown in seqkit seems to be related to file access? Large sys time. Can you add a link on how to download the benchmark data so I can compare on my machine and try rg?
|
|
| May 31, 2022 at 17:42 | comment | added | M__♦ | @CorneliusRoemer you've got to agree with terdon: seqkit is far too slow. It is worrying. This would cause problems on the data sets in question where a million strong fasta file is normal. Take your pick but realistically its the end of this discussion, there's no point using an inefficient algorithm against huge data sets, sooner or later they'll be a RAM issue. | |
| May 31, 2022 at 17:11 | history | edited | terdon♦ | CC BY-SA 4.0 |
added 1230 characters in body
|
| May 31, 2022 at 15:05 | comment | added | terdon♦ | Heh, you see how memorable I find it, @CorneliusRoemer! :P | |
| May 31, 2022 at 15:03 | comment | added | Cornelius Roemer |
Fair :) though it's seqkit not seqt
|
|
| May 31, 2022 at 14:51 | comment | added | terdon♦ |
@CorneliusRoemer answers providing different approaches are welcome across the Stack Exchange network (note that this site is not Stack Overflow). Especially when there is already a fine answer with using the requested tool. As for what is memorable, that is entirely personal. You are familiar with seqt so that seems easy and memorable for you. I have been using *nix tools for 20 years, so nothing is more memorable than a simple sed s/foo/bar/. But that's fine! That's the beauty of it: we can all provide answers and people can choose whichever approach works well for them. Everybody wins!
|
|
| May 31, 2022 at 14:34 | comment | added | Cornelius Roemer |
@M__ of course there are use cases for *nix tools. But there are also use cases for *kits. In this case I wanted to use a memorable solution that will not require lookup every time. The two other answers are arguably off topic because they don't address the question - which explicitly asked for seqkit usage. Another *kit would have been more on-topic than *nix tools. I don't mind them, it's just a bit typical for SO, you ask a question and you get a solution that's for a different question ;)
|
|
| May 31, 2022 at 14:17 | comment | added | M__♦ | @CorneliusRoemer there are weakness on reliance on a developers 'kit'. Flexibility, if you wanted to count the number of pangotype A.1 in SARS-CoV-2, my code is easy to modify to />.*A\.1.*/ and introduce $count++ , whereas you would need to search seqkit to see if the developer had foresight. Bugs: are 'kits' completely bug free, how can you be sure? Upgrades: the promised upgrade ain't ever released (very, very common). Distributing your algorithm: this is a headache because you've now got a dependency which you don't have permission to redistribute. | |
| May 31, 2022 at 8:35 | history | edited | terdon♦ | CC BY-SA 4.0 |
added 5 characters in body
|
| May 31, 2022 at 8:34 | comment | added | terdon♦ | @CorneliusRoemer I guess. I have never used it myself and have been doing bioinformatics for more than 20 years. But by "standard" I mean "standard *nix tools", not bioinformatics tools. I edited to clarify. | |
| May 30, 2022 at 20:08 | comment | added | Cornelius Roemer | Seqkit is arguably a bioinfo standard tool ;) | |
| May 30, 2022 at 17:49 | history | answered | terdon♦ | CC BY-SA 4.0 |