How long should it take to cluster Uniref?

Question

I have the Uniref90 dataset containing around 76M sequences. I have made modifications to all sequences based on a reduced amino acid alphabet.

I am trying to cluster it again at 90% identity. I am using mmseq cluster on a 64 core, 500GB RAM machine. It has been more than 12h now.

Starting prefiltering scores calculation (step 1 of 1)
Query db start 1 to 74390789
Target db start 1 to 74390789
[=

How long does it normally take to cluster such a database? I know I can use linclust instead to cluster in linear time, but it might not be what I need in this case.

I am using mmseq cluster as follows:

mmseqs createdb "uniref90-$1.fasta" "uniref90-DB-$1"
mmseqs cluster "uniref90-DB-$1"  "uniref90-DB-$1_clu" tmp/ --cov-mode 2 -c 0.8 --min-seq-id 0.9

Answer 1

When I want to estimate the time taken to do something big, I subset my input data to something that takes a few minutes or less, and calculate forward from there. If there's alignment involved, it's usually a good idea to work it out for multiple dataset sizes, as there could be a linear relationship between input size and time taken, square relationship, or worse.

Have you looked at how long it takes for, say, 1000 sequences, 2000 sequences, and 5000 sequences?

Answer 2

The mmseqs2 code should be specifically compiled and optimised for your system. mmseqs2 optimisation is automated and might be available via Spack* (I'm not sure its actually done for mmseqs2), it is definitely available for EasyBuild. I wouldn't advise attempting a manual optimisation unless you've prior experience.

How much time this will save is more difficult to assess but 50% faster is a conservative assessment. This approach is good practice for a job like this size and its good for the associated report, its standard HPC practice. If you're doing cloud computation (i.e. $$$), HPC optimisation is essential. AWS make serious money from suboptimal HPC practice (in fairness to them they also do a lot of development work to minimise costs).

Finally, if you're an expert in optimisation you can save serious computational time, but this is a major investment of your personal development time and technique. The rationale of how and why is complicated.

*, its very likely available on AWS.

---

You might simply consider its been running for a few days already so HPC technique isn't needed for this specific job. However, you'll know for the future.