# Help to create a dataframe in Python from a FASTA file

I want to create a dataframe in Python starting from a FASTA format file. Given the toy FASTA file that I am attaching, I built this program in Python that returns four colums corresponding to id, sequence length, sequence, animal name and rows corresponding to all the data available.

However, I am trying to understand how to modify this code in order to create a dataframe in which classes Human and Dog have the same number of data. For example, I want to say to Python: "Append to record (that is the empty list) id, sequence length, sequence and animal for Human, but do it a number of times that is equal to the number of data of the class with minimum number of data (that is Dog)".

I think that a while loop is needed but I have a bit troubles to understand how to do it. Any suggestion ?

Below the Python code I wrote and the FASTA format file I used.

import pandas as pd
import re
from Bio.SeqIO.FastaIO import SimpleFastaParser
with open("Proof.txt") as fasta_file :
records = [] # create empty list
for title, sequence in SimpleFastaParser(fasta_file): #SimpleFastaParser Iterate over Fasta records as string tuples. For each record a tuple of two strings is returned, the FASTA title line (without the leading ‘>’ character), and the sequence (with any whitespace removed).
record = []
title_splits=re.findall(r"[\w']+", title) # Data cleaning is needed

record.append(title_splits[0])  #First values are ID (Append adds element to a list)
record.append(len(sequence)) #Second values are sequences lengths
sequence = " ".join(sequence) #It converts into one line
record.append(sequence)#Third values are sequences

#Fourth column will contain the species
if "Human" in title_splits:
record.append("Human")
else:
record.append("Dog")

records.append(record)
return pd.DataFrame(records, columns = columns) #We have created a function that returns a dataframe

#Now let's use this function by inserting in the first argument the file name (or file path if your working directory is different from where the fasta file is)
#And in the second one the names of columns
data = read_fasta("Proof.txt", columns=["id","sequence_length", "sequence", "animal"])
data


The FASTA format file is this:

>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK


My code prints a dataframe like:

       id  sequence_length                               sequence animal
0   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
1   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
2   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
3   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
4   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
5   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
6   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
7   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
8   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
9   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
10  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
11  Numer               19  H S S F I E I V N I E H V I E H I V K  Human


But I would like that the number of rows for Human is the same for Dog (because, in other words, I would like the same number of data for each class).

Hoping to have been clear, I thank you in advance.

P.S.: My program is an adapted version of this notebook .

• Please correct me if I am wrong, you want to print the lines Human 4 times because it is equal to the number of rows that corresponds to Dog, is that right? Jan 15 at 22:39

Assuming what I said in the comments is true. Here's my attempt, there may be an easier solution but this should do it:

import pandas as pd
import re

# a function to return the count number of animals in the fasta in a dictionary
def count_animals(file_path, separator, animal_index):
animal_dictionary={}
with open("Proof.txt") as fasta_file:
for line in fasta_contents:
if line.startswith(">"):
animal = line.split(separator)[animal_index]
if animal not in animal_dictionary:
animal_dictionary[animal] = 1
else:
animal_dictionary[animal] = animal_dictionary[animal] + 1
return animal_dictionary

counter = 0  # counter used to print no more than number of times 'Dog'
animal_d = count_animals(file_path, '|', 1) # execute function
max_animal =  max(animal_d.values()) # get max number of animals
min_animal =  min(animal_d.values()) # get min number of animals

from Bio.SeqIO.FastaIO import SimpleFastaParser
with open("Proof.txt") as fasta_file :
records = [] # create empty list
for title, sequence in SimpleFastaParser(fasta_file): #SimpleFastaParser Iterate over Fasta records as string tuples. For each record a tuple of two strings is returned, the FASTA title line (without the leading ‘>’ character), and the sequence (with any whitespace removed).
record = []
title_splits=re.findall(r"[\w']+", title) # Data cleaning is needed

record.append(title_splits[0])  #First values are ID (Append adds element to a list)
record.append(len(sequence)) #Second values are sequences lengths
sequence = " ".join(sequence) #It converts into one line
record.append(sequence)#Third values are sequences
# if 'Human' exists in title and the counter is less than min_animal('Dog'), add the record
if 'Human' in title and counter < min_animal:
counter += 1 # add +1 to counter

#Fourth column will contain the species
record.append("Human")

records.append(record)
elif 'Dog' in title:
record.append("Dog")

records.append(record)

# print(records)
return pd.DataFrame(records, columns = columns) #We have created a function that returns a dataframe

#Now let's use this function by inserting in the first argument the file name (or file path if your working directory is different from where the fasta file is)
#And in the second one the names of columns
data = read_fasta("Proof.txt", columns=["id","sequence_length", "sequence", "animal"])

print(data)


Should give:

      id  sequence_length                               sequence animal
0  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
1  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
2  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
3  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
4  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
5  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
6  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
7  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog

• Thank you very much ! Jan 18 at 15:54

An example of contructing a pandas dataframe for fasta on Stack Exchange is here . This is clean code deploying zip outside Biopython. Constructing a pandas dataframe directly via dictionary.

Are you intending to use split method directly on the sequence, such that each residue is a separate column in a dataframe?

Anyway,

title_splits=re.findall(r"[\w']+", title) # Data cleaning is needed


This is a bad idea because you will remove the Genbank unique identifier (its an integer) and this should not leave the sequence.

You will need to switch out of the Fasta object. Your approacha and @user324810's use of

sequence = " ".join(sequence)


is okayish. However, I think it should be '', or "" (no space character). I spotted manually removing fasta, well the method should do this automatically. I am not familiar with Fastaparser, but AlignIO and SeqIO will both automatically process fasta without the need deal with ">". Biopython will have a length method and whilst you have the Biopython object it is preferable to use Biopython (OOP) methods.

from Bio.SeqIO.FastaIO import SimpleFastaParser


This should be at the top of the code.

Overall, you manually open the data, pass it through the Bioparser then immediately dump the object. I do agree to write a pandas dataframe from a Biopython object these conversion to string, but Biopython will provide a method to print the object (pass it to a string) without all the parsing. Some of the parsing is removing key data (Genbank ID).

Personally I would look at the Stackoverflow example and base your code around this.

You clearly have an understanding of pandas, but I get the impression Biopython is something your not familiar with. My suspicion is that Biopython would infact answer the entirety of your question (whatever it might be) without passing to pandas.

pandas is cool and powerful for non-molecular data, but unless your using machine/deep learning (where a dataframe is necessary input) I think Biopython will provide most of the answers.

Important Terminology The class is FastaIO and the method is SimpleFastaParser. Humans and dogs are instances of this method and are therefore object(s).

Please don't write 'class' of dogs and humans in a formal report because it is incorrect OOP and will cause alarm if your reviewer is a coder. Your code is functional programming rather than OOP. You could say the human/dog fasta object, or just refer to it as human/dog dataframes and avoid any OOP terminology.

• Thank you for the suggestions @M__ but SimpleFastaParser does not remove Genbank ID... or at least the FASTA file that I have is with the header like this: >Numer|Animal|OtherInformations . And once SimpleFastaParser, for each record, has returned a tuple of two strings (and the header that is without ">"), re.findall has splitted the header string according to all characters (or at least according to the "|") that I am interested in, I can use this splitting to identify the name of the species and append it to the empty list... Jan 18 at 16:41
• The only thing is in fact that re.findall splits also according to point as separator and I noticed that GenBank IDs are numbers like NP288364.1 ... So re.findall will split NP288364.1 into NP288364 and 1 . Is it so terrible ? I found that the ".1" indicates the "version" of the sequence. Do you know a way to change re.findall so that I can split without considering the points ? Jan 18 at 16:44
• Yes Biopython is new for me but it does not delete the ID... maybe the fault lies with re.findall, isn't it ? Jan 18 at 16:46
• Yes, you should use different parsing such as re.sub explicitly stating "|" such as using a capture "(.+)|.*" and \$1, or \1 in re.sub or using or re.final ([\w|\d], string).
– M__
Jan 18 at 18:38

Just to add, usually the way to move a sequence object to a neat, decluttered string in Biopython is not to do this manually as done here:

title_splits=re.findall(r"[\w']+", title)
sequence = " ".join(sequence)


both being from SimpleFastaParser(fastafile)

It should be possible and, if so, better to state,

 str(title)
str(sequence)


If you can issue a print command and the string appears nice and neat on the screen str() will work.

This becomes important for complex sequence/alignment objects, such as SeqIO and AlignIO. The reason it works is __str__ has been declared or available through OOP inheritance.

• Thank you, however in fact I realize only now that sequence = " ".join(sequence) is not necessary since SimpleFastaParser already joins sequences into one line and as regards the header I tried with str as you say but it does not work because it leaves it as it is Numer|Human|Hearth and I do not know how to say "append Number". I could write like title_str=str(title) record.append(title_str[0:6]) but the length of the id number varies for each header... Jan 28 at 9:27
• Fine good to know. str isn't implemented for this given method (they haven't written one). It is a standard method in most OOP, but not all
– M__
Jan 28 at 16:34