Currently algorithmically developing a calculation for "co-dependent" mutation (quotation appropriate). The algorithm uses chunked biopython alignment objects which are parallelized.

Aim One optization is caching a database within parallelization, but I ain't sure whether the parallelization duplicates the cache.

Method Each alignment chunk (biopython object) is passed onto a separate CPU and refers back to the original database to obtain its metadata in the final stage of the calculation (dragging metadata around this algorithm makes no sense).

The idea is to cache the database to prevent pulling it in from the harddrive:

from multiprocessor import Pool
from myOOPlibrary import mystuff    

Facade design, the parallelised index to the alignment chunks enter into the myOOPlibrary:

from cachetools import cached, TTLCache

cache = TTLCache(maxsize=1, ttl=500)

Class mystuff (stuff2):
... # init omitted

    def mymethod(self, ..):
    # to be updated 
Class stuff2 ():
... # init omitted

    def getDB(self, ...):
.... #  pull it in (harddrive) and parse to a dictionary
    return metadataDB

Question I'm wondering if caching is the right approach in parallelization, specifically whether this will end up with multiplying the database held in RAM...

Alternatives The alternativees to caching are:

  • IO to JSON (not cool)
  • IO via pickle (binary IO): a JSON will readilypickle

The possible advantage of the alternativees are the database is written once and accessed by all cores for certain.

Underlying question

  • When caching is the database cached for each core? If so the RAM overhead becomes an issue because the database is duplicated total CPUs - 1 times
  • Alternatively is the cache shared between CPUs centrally and each CPU refers to this location?
  • 2
    $\begingroup$ As currently written, this might be more appropriate for Stack Overflow $\endgroup$ Jun 28, 2022 at 13:40
  • $\begingroup$ Possibly but parallelization of biopython objects must be fairly common in algorithm/pipeline development. Alot work with biopython here $\endgroup$
    – M__
    Jun 28, 2022 at 14:32
  • 1
    $\begingroup$ What are you using for the parallelization part? If it's just the built-in threading module, I don't think anything will be duplicated there, you just have to make sure you don't step on your own toes with any read/write aspects (e.g. use something like Queue objects if needed). But in that case, why using a caching library at all, versus just a single shared DB object you load once? (More of the rest of the code might help clarify.) $\endgroup$
    – Jesse
    Jun 28, 2022 at 15:59
  • $\begingroup$ Updated, @Jesse comment is cool and closes in on an easy solution. $\endgroup$
    – M__
    Jun 28, 2022 at 23:28
  • 1
    $\begingroup$ If python code is the bottleneck, rewriting in C/C++ may get you a 50-fold speedup. $\endgroup$
    – user172818
    Jun 28, 2022 at 23:37

1 Answer 1



  1. @Jesse's comment I think is the right answer, open the database outside parallelisation. What I didn't thoroughly describe (and what was essentially requested) is the architecture, the alignments entering into an entire OOP library and the whole library is parallelised, i.e. every new alignment entering library is on its own CPU. The answer is obvious: instantiate the OOP library with the entire database and thats a key purpose of the design.
  2. The technical answer is joblib. This is built to cache across parallelised archetechture as the joblib team point out here. The criticism is that there's parallelisation within parallelisation. The discussion* is as much in the comments as in the response.
  3. @user172818 response is correct too, the algorithm would benefit from C. Its a lot of code to rewrite in C, but sidestepping via Cython is a future development (its a hybrid language connecting C and Python).

*, https://stackoverflow.com/questions/25033631/multiple-processes-sharing-a-single-joblib-cache

There was a comment hinting this is Stack Overflow, that's difficult. Ultimately the technical answer (which is Stack Overflow) isn't the correct answer as far as I understand OOP.


Just to wrap this, point 1. worked fine, thank you. It means a database sits on each processor in parallelisation, but is imported once. In truth, it worked nicely and provided cleaner code. Re-writing to instantiate once would affect other stuff so would be more costly than a cost-saving.

Python can be very fast (e.g. numpy), but also really slow (hence it has critiques), extensively parallelising helps bridge the gap. Python parallelisation has critiques which I strongly, strongly believe are unfounded: I parallelise loads via multiprocessor library no problems.

Caching here, i.e. parallelising across a parallelisation could stretch things too far.

Last point

Point 3, re-writing in Cython/C, is a greater priority than the testing required for parallelised caching: can't parallelise all data and that code is better as C.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.