0
$\begingroup$

I have the Uniref90 dataset containing around 76M sequences. I have made modifications to all sequences based on a reduced amino acid alphabet.

I am trying to cluster it again at 90% identity. I am using mmseq cluster on a 64 core, 500GB RAM machine. It has been more than 12h now.

Starting prefiltering scores calculation (step 1 of 1)
Query db start 1 to 74390789
Target db start 1 to 74390789
[=

How long does it normally take to cluster such a database? I know I can use linclust instead to cluster in linear time, but it might not be what I need in this case.

I am using mmseq cluster as follows:

mmseqs createdb "uniref90-$1.fasta" "uniref90-DB-$1"
mmseqs cluster "uniref90-DB-$1"  "uniref90-DB-$1_clu" tmp/ --cov-mode 2 -c 0.8 --min-seq-id 0.9
$\endgroup$
5
  • $\begingroup$ Thanks. Hmmm ... when you use htop what do you see when its running? $\endgroup$
    – M__
    Nov 21, 2023 at 0:25
  • $\begingroup$ CPU is used at the maximum and 20% of RAM. Anyway, I have looked into the mmseqs2 in more detail and found that it should take at least a few days. $\endgroup$
    – GIONII
    Nov 21, 2023 at 11:57
  • $\begingroup$ Thanks and thank you for your question. Thats very clear and answered below. $\endgroup$
    – M__
    Nov 21, 2023 at 13:07
  • $\begingroup$ What does "CPU is used at the maximum" mean? Do you see 100% in the CPU% column? Or do you see 6400%? $\endgroup$
    – terdon
    Nov 21, 2023 at 13:21
  • $\begingroup$ 6400 %CPU, 12.8 %MEM $\endgroup$
    – GIONII
    Nov 22, 2023 at 13:17

2 Answers 2

0
$\begingroup$

When I want to estimate the time taken to do something big, I subset my input data to something that takes a few minutes or less, and calculate forward from there. If there's alignment involved, it's usually a good idea to work it out for multiple dataset sizes, as there could be a linear relationship between input size and time taken, square relationship, or worse.

Have you looked at how long it takes for, say, 1000 sequences, 2000 sequences, and 5000 sequences?

$\endgroup$
0
$\begingroup$

The mmseqs2 code should be specifically compiled and optimised for your system. mmseqs2 optimisation is automated and might be available via Spack* (I'm not sure its actually done for mmseqs2), it is definitely available for EasyBuild. I wouldn't advise attempting a manual optimisation unless you've prior experience.

How much time this will save is more difficult to assess but 50% faster is a conservative assessment. This approach is good practice for a job like this size and its good for the associated report, its standard HPC practice. If you're doing cloud computation (i.e. $$$), HPC optimisation is essential. AWS make serious money from suboptimal HPC practice (in fairness to them they also do a lot of development work to minimise costs).

Finally, if you're an expert in optimisation you can save serious computational time, but this is a major investment of your personal development time and technique. The rationale of how and why is complicated.

*, its very likely available on AWS.


You might simply consider its been running for a few days already so HPC technique isn't needed for this specific job. However, you'll know for the future.

$\endgroup$
2
  • 1
    $\begingroup$ Is the tool multithreaded? The OP mentioned "CPU is used at the maximum" in htop which makes me think that it might be 100% of a single core. $\endgroup$
    – terdon
    Nov 21, 2023 at 13:16
  • $\begingroup$ @terdon. Yes it is. Whether it's default - dunno - I was thinking that too. Which is one reason why I requested the htop, but OPs response strongly suggests the parallelisation is default. It's easy to code parallelisation as default (at least in Python), normally its Total CPU - 1. What is a bit unusual I'll admit ... one CPU should be free (standard coding practice) but the OP didn't report that. However, we don't know how the original coder did this. To be honest mmseqs coder is hot, so they'd know the convention. If mmseqs2 is intended for HPC Total CPU - 1 is dropped $\endgroup$
    – M__
    Nov 21, 2023 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.