Currently algorithmically developing a calculation for "co-dependent" mutation (quotation appropriate). The algorithm uses chunked biopython alignment objects which are parallelized.
Aim One optization is caching a database within parallelization, but I ain't sure whether the parallelization duplicates the cache.
Method Each alignment chunk (biopython object) is passed onto a separate CPU and refers back to the original database to obtain its metadata in the final stage of the calculation (dragging metadata around this algorithm makes no sense).
The idea is to cache the database to prevent pulling it in from the harddrive:
from multiprocessor import Pool from myOOPlibrary import mystuff
Facade design, the parallelised index to the alignment chunks enter into the myOOPlibrary:
from cachetools import cached, TTLCache cache = TTLCache(maxsize=1, ttl=500) Class mystuff (stuff2): ... # init omitted def mymethod(self, ..): # to be updated Class stuff2 (): ... # init omitted @cached(cache) def getDB(self, ...): .... # pull it in (harddrive) and parse to a dictionary return metadataDB
Question I'm wondering if caching is the right approach in parallelization, specifically whether this will end up with multiplying the database held in RAM...
Alternatives The alternativees to caching are:
- IO to JSON (not cool)
- IO via pickle (binary IO): a JSON will readily
The possible advantage of the alternativees are the database is written once and accessed by all cores for certain.
- When caching is the database cached for each core? If so the RAM overhead becomes an issue because the database is duplicated
total CPUs - 1times
- Alternatively is the cache shared between CPUs centrally and each CPU refers to this location?