I need help with my Biopython code, I am kind of new to Python and Biopython so please don't be harsh on me.
Everything started from here: Remove duplicated sequences in FASTA with Python
I answered copying code from here: Remove Redundant Sequences from FASTA file in Python but I am still bugged about the OP's first question: Remove duplicated sequences in FASTA with Python
So I decided to try to minimize the code footprint on RAM. I ended up copying from Biopython documentation Split large file and producing:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
Created on Thu Nov 11 17:40:03 2020
@author: Pietro
"""
from memory_profiler import memory_usage
import memory_profiler
def batch_iterator(iterator, batch_size):
""" from https://biopython.org/wiki/Split_large_file
"""
entry = True # Make sure we loop once
while entry:
batch = []
while len(batch) < batch_size:
try:
entry = iterator.__next__()
except StopIteration:
entry = None
if entry is None:
# End of file
break
batch.append(entry)
if batch:
yield batch
# @profile
def f():
from Bio import SeqIO
import time
import sys
start = time.time()
data_input_iter = SeqIO.parse("Newtextfile-03", "fasta")
print('data_input_iter : ',sys.getsizeof(data_input_iter),'\n')
c=0
for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
record_iter = batch
val = True
c += 1
if c == 1:
with open("removed-double_Newtextfile-03", 'w+') as output_file:
SeqIO.write(record_iter, output_file, "fasta")
# print('copy : ',(record_iter))
output_file.flush()
output_file.close()
else:
SeqIO.write(record_iter, "temp_file", "fasta")
record_iter = SeqIO.read("temp_file", "fasta")
data_output_iter = SeqIO.parse("removed-double_Newtextfile-03", "fasta")
d = 0
try:
for i , batch in enumerate(batch_iterator(data_output_iter , 1)):
SeqIO.write(batch, "temp_file2", "fasta")
record_output = SeqIO.read("temp_file2", "fasta")
d += 1
if record_iter.seq != record_output.seq :
# print('pass : ',record_iter.name)
pass
else:
val = False
# print('false')
break
except:
print('ERROR !!!!!!!!!!!')
quit()
if val:
# print('copy : ',record_iter.name)
with open("removed-double_Newtextfile-03", 'a+') as output_file:
output_file.write(str(record_iter.format("fasta")))
output_file.close()
del data_output_iter
end = time.time()
print(f"Run time is {(end- start)}")
mem_usage = memory_usage(f)
# print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))
# f()
but I have to major problems :
FIRST: I wasn't able to find a way to pass a SeqRecord object from a list to a file without saving it to a temp file first. That is without:
for i , batch in enumerate(batch_iterator(data_input_iter , 1)):
record_iter = batch
......
SeqIO.write(record_iter, "temp_file", "fasta")
record_iter = SeqIO.read("temp_file", "fasta")
the line:
if record_iter.seq != record_output.seq :
will throw:
AttributeError: 'list' object has no attribute 'format'
AttributeError: 'list' object has no attribute 'seq'
you get first error and the if remove format from:
output_file.write(str(record_iter.format("fasta")))
and #quit() you get the second error How should I cope with this ? I mean is there a way to bypass the save and read lines ?
SECOND : and more important. I am sure my code doesn't use a set and a list .... and that's why is extremely slow compared to the copied Remove Redundant Sequences from FASTA file in Python posted below too. But how does it use even more RAM that the fastest code ? I tried to minimize the memory footprint but wasn't able !! Am I using a bad way to try to calculate the amount of RAM used ? And does Biopythom SeqIO.parse object already is the optimal choice for , lets say "2/4 Gb size" fasta files on a laptop with just 4Gb of RAM (remember the original OP post Remove duplicated sequences in FASTA with Python). Maybe my test file is not big enough ? Does anybody have a 2GB fasta file with a dozen of duplicates in it ?
Remove duplicated sequences in FASTA with Python memory usage:
size of parser : 48
memory seen : 32984
memory records : 2888
Run time is 0.0008047382036844889
Maximum memory usage: 55.359375
my code memory usage and speed:
data_input_iter : 48
Run time is 37.781240463256836
Maximum memory usage: 56.40234375
My fasta test set is: https://easyupload.io/z2yss1
Biopython method to remove duplicated from Remove duplicated sequences in FASTA with Python:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
"""
copied from https://bioinformatics.stackexchange.com/questions/13708/remove-redundant-sequences-from-fasta-file-in-python
uses set() instead of list for seen
should work faster for big files
"""
from memory_profiler import memory_usage
def f():
import sys
from Bio import SeqIO
import time
start = time.time()
seen = set()
records = []
parser = SeqIO.parse("Newtextfile-03", "fasta")
for record in parser:
# print('running')
if str(record.seq) not in seen:
seen.add(str(record.seq))
records.append(record)
# print('copying')
print('size of parser : ',sys.getsizeof(parser) )
print('memory seen : ',sys.getsizeof(seen))
print('memory records : ',sys.getsizeof(records))
#writing to a fasta file
SeqIO.write(records, "removed-double_Newtextfile-03-from-002", "fasta")
end = time.time()
print(f"Run time is {(end- start)/60}")
mem_usage = memory_usage(f)
# print('Memory usage (in chunks of .1 seconds): %s' % mem_usage)
print('Maximum memory usage: %s' % max(mem_usage))
# f()
