0
$\begingroup$

I ran into the following error when using cooler to load a contact pairs file:

ubuntu@ip-172-31-8-42:/data/hg-tmp$ FILE=GSE109344_monkey_fibro_allValidPairs.txt.gz; sudo docker run -v $(pwd):/data $TAG cooler cload pairs -c1 2 -p1 3 -c2 5 -p2 6 /data/chromInfo.rheMac2.
simple.txt:${BINSIZE} /data/$FILE /data/${FILE}.cool;
Traceback (most recent call last):
  File "/usr/local/bin/cooler", line 11, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python2.7/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/cooler/cli/cload.py", line 477, in pairs
    map(pipeline, reader),
  File "cytoolz/functoolz.pyx", line 493, in cytoolz.functoolz.Compose.__call__
  File "/usr/local/lib/python2.7/site-packages/cooler/create/_ingest.py", line 425, in _aggregate_records
    .aggregate(agg)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 1315, in aggregate
    return super(DataFrameGroupBy, self).aggregate(arg, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 186, in aggregate
    result, how = self._aggregate(arg, _level=_level, *args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 498, in _aggregate
    result = _agg(arg, _agg_1dim)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 449, in _agg
    result[fname] = func(fname, agg_how)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/base.py", line 432, in _agg_1dim
    return colg.aggregate(how, _level=(_level or 0) + 1)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/generic.py", line 760, in aggregate
    return getattr(self, func_or_funcs)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/groupby.py", line 1227, in size
    result = self.grouper.size()
  File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 225, in size
    ids, _, ngroup = self.group_info
  File "pandas/_libs/properties.pyx", line 36, in pandas._libs.properties.CachedProperty.__get__
  File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 252, in group_info
    comp_ids, obs_group_ids = self._get_compressed_labels()
  File "/usr/local/lib/python2.7/site-packages/pandas/core/groupby/ops.py", line 272, in _get_compressed_labels
    return compress_group_index(group_index, sort=self.sort)
  File "/usr/local/lib/python2.7/site-packages/pandas/core/sorting.py", line 379, in compress_group_index
    comp_ids, obs_group_ids = table.get_labels_groupby(group_index)
  File "pandas/_libs/hashtable_class_helper.pxi", line 1255, in pandas._libs.hashtable.Int64HashTable.get_labels_groupby
  File "pandas/_libs/hashtable_class_helper.pxi", line 199, in pandas._libs.hashtable.Int64Vector.resize
MemoryError: cannot allocate memory for array

The machine has 8Gb of RAM. Any ideas?

$\endgroup$
5
$\begingroup$

It seems to be some kind of a memory leak in pandas's read_csv, maybe due to it doing gzip-decompression.

Doing the decompression in a separate process and piping the result into cooler cload pairs should do the trick.

gzip -dc GSE109344_monkey_fibro_allValidPairs.txt.gz | cooler cload pairs -c1 2 -p1 3 -c2 5 -p2 6 /data/chromInfo.rheMac2.
simple.txt:${BINSIZE} - /data/${FILE}.cool
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.