0
$\begingroup$

WGCNA carefully takes advantage of server parallelization in several functions, and I can switch this on with:

enableWGCNAThreads(nThreads=28)

That speeds up some functions significantly, but the two bottleneck functions in the pipeline (in my experience) are part of the "Projective k-means" operation, primarily "..k-means clustering.." and "..merging smaller clusters..".

Unfortunately, parallelization does not work for those two functions.

Am I missing something? How can I parallelize these functions? Otherwise I'm wasting HPC resources and cluster time.

For reference, I'm working on a 28-processor HPC node with 96GB RAM, and I'm trying to run analysis on a 15,000-square correlation matrix.

$\endgroup$
2
$\begingroup$

The bad news is that, indeed, projectiveKMeans is not parallelized and I am not sure how much of it is (easily) parallelizable. The good news is that with 15k features (genes) and 96GB RAM you don't need to run preclustering at all. Just analyze the whole data set in one block.

If you for some reason insist on splitting the data into blocks, make sure the BLAS R uses on the cluster is a fast BLAS, ideally one that can parallelize matrix operations. A lot of time in merging of small clusters is spent calculating singular value decompositions, which becomes much faster using a good BLAS. HPC clusters will often have Intel KML available, or the admins could install OpenBLAS.

$\endgroup$
  • $\begingroup$ Thanks for each suggestion. Both helpful! I definitely don't need to run preclustering, makes sense. Will also make sure we are running Intel KML or OpenBLAS, I suspect this will already have been considered and optimized. $\endgroup$ – sissypants Dec 24 '19 at 13:06
  • $\begingroup$ Will WGCNA k-means run using GPU? I have a hunch it might do. $\endgroup$ – M__ Dec 24 '19 at 13:43
  • $\begingroup$ Okay I checked and the answer is no. You're stuck, my bad. Reference bmcsystbiol.biomedcentral.com/articles/10.1186/… $\endgroup$ – M__ Dec 24 '19 at 13:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.