Remove Redundant Sequences from FASTA file in Python

Question

I'm attempting to remove redundant sequences from a fasta file (from NCBI). When I execute this code, it returns the number of spots, not the number of sequences. (Number of spots: 408,293, Number of sequences: 40,830)

Please let me know if you have run a code that can remove the duplicate ideas or have any ideas/suggestions.

Thank you SO much.

from Bio import SeqIO
import time

start = time.time() 

seen = []
records = []

for record in SeqIO.parse("b4r2.fasta", "fasta"):  
    if str(record.seq) not in seen:
        seen.append(str(record.seq))
        records.append(record)


#writing to a fasta file
SeqIO.write(records, "no_dupes_b4r2.fasta", "fasta")
end = time.time()

print(f"Run time is {(end- start)/60}") 

Answer 1

Your code (as currently formatted) should generate the correct output, but it is not efficient. Make seen a set to improve these O(n) in calls to O(1) time complexity- this will speed it up a lot

seen = set()
records = []

for record in SeqIO.parse("b4r2.fasta", "fasta"):  
    if record.seq not in seen:
        seen.add(record.seq)
        records.append(record)