Help to create a dataframe in Python from a FASTA file

Question

I want to create a dataframe in Python starting from a FASTA format file. Given the toy FASTA file that I am attaching, I built this program in Python that returns four colums corresponding to id, sequence length, sequence, animal name and rows corresponding to all the data available.

However, I am trying to understand how to modify this code in order to create a dataframe in which classes Human and Dog have the same number of data. For example, I want to say to Python: "Append to record (that is the empty list) id, sequence length, sequence and animal for Human, but do it a number of times that is equal to the number of data of the class with minimum number of data (that is Dog)".

I think that a while loop is needed but I have a bit troubles to understand how to do it. Any suggestion ?

Below the Python code I wrote and the FASTA format file I used.

import pandas as pd
import re
def read_fasta(file_path, columns) :
    from Bio.SeqIO.FastaIO import SimpleFastaParser 
    with open("Proof.txt") as fasta_file :  
        records = [] # create empty list
        for title, sequence in SimpleFastaParser(fasta_file): #SimpleFastaParser Iterate over Fasta records as string tuples. For each record a tuple of two strings is returned, the FASTA title line (without the leading ‘>’ character), and the sequence (with any whitespace removed). 
            record = []
            title_splits=re.findall(r"[\w']+", title) # Data cleaning is needed
          
                 
            
            record.append(title_splits[0])  #First values are ID (Append adds element to a list)
            record.append(len(sequence)) #Second values are sequences lengths
            sequence = " ".join(sequence) #It converts into one line
            record.append(sequence)#Third values are sequences
                
            #Fourth column will contain the species
            if "Human" in title_splits:
                    record.append("Human")    
            else:
                    record.append("Dog")
                
           
              
            records.append(record)
    return pd.DataFrame(records, columns = columns) #We have created a function that returns a dataframe

#Now let's use this function by inserting in the first argument the file name (or file path if your working directory is different from where the fasta file is)        
#And in the second one the names of columns
data = read_fasta("Proof.txt", columns=["id","sequence_length", "sequence", "animal"])
data

The FASTA format file is this:

>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Dog|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK
>Numer|Human|Hearth
HSSFIEIVNIEHVIEHIVK

My code prints a dataframe like:

       id  sequence_length                               sequence animal
0   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
1   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
2   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
3   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
4   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
5   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
6   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
7   Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
8   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
9   Numer               19  H S S F I E I V N I E H V I E H I V K  Human
10  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
11  Numer               19  H S S F I E I V N I E H V I E H I V K  Human

But I would like that the number of rows for Human is the same for Dog (because, in other words, I would like the same number of data for each class).

Hoping to have been clear, I thank you in advance.

P.S.: My program is an adapted version of this notebook .

Answer 1

Assuming what I said in the comments is true. Here's my attempt, there may be an easier solution but this should do it:

import pandas as pd
import re

# a function to return the count number of animals in the fasta in a dictionary
def count_animals(file_path, separator, animal_index):
  animal_dictionary={}
  with open("Proof.txt") as fasta_file:
    fasta_contents = fasta_file.readlines()
    for line in fasta_contents:
      if line.startswith(">"):
        animal = line.split(separator)[animal_index]
        if animal not in animal_dictionary:
          animal_dictionary[animal] = 1
        else:
          animal_dictionary[animal] = animal_dictionary[animal] + 1
  return animal_dictionary

def read_fasta(file_path, columns) :
  counter = 0  # counter used to print no more than number of times 'Dog'
  animal_d = count_animals(file_path, '|', 1) # execute function
  max_animal =  max(animal_d.values()) # get max number of animals
  min_animal =  min(animal_d.values()) # get min number of animals

  from Bio.SeqIO.FastaIO import SimpleFastaParser 
  with open("Proof.txt") as fasta_file :
    records = [] # create empty list
    for title, sequence in SimpleFastaParser(fasta_file): #SimpleFastaParser Iterate over Fasta records as string tuples. For each record a tuple of two strings is returned, the FASTA title line (without the leading ‘>’ character), and the sequence (with any whitespace removed). 
      record = []
      title_splits=re.findall(r"[\w']+", title) # Data cleaning is needed

      record.append(title_splits[0])  #First values are ID (Append adds element to a list)
      record.append(len(sequence)) #Second values are sequences lengths
      sequence = " ".join(sequence) #It converts into one line
      record.append(sequence)#Third values are sequences
      # if 'Human' exists in title and the counter is less than min_animal('Dog'), add the record
      if 'Human' in title and counter < min_animal:
        counter += 1 # add +1 to counter

        #Fourth column will contain the species
        record.append("Human")  
          
        records.append(record)
      elif 'Dog' in title:
        record.append("Dog")  
          
        records.append(record)

    # print(records)
  return pd.DataFrame(records, columns = columns) #We have created a function that returns a dataframe

#Now let's use this function by inserting in the first argument the file name (or file path if your working directory is different from where the fasta file is)    
#And in the second one the names of columns
data = read_fasta("Proof.txt", columns=["id","sequence_length", "sequence", "animal"])

print(data)

Should give:

      id  sequence_length                               sequence animal
0  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
1  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
2  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
3  Numer               19  H S S F I E I V N I E H V I E H I V K  Human
4  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
5  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
6  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog
7  Numer               19  H S S F I E I V N I E H V I E H I V K    Dog

Answer 2

An example of contructing a pandas dataframe for fasta on Stack Exchange is here . This is clean code deploying zip outside Biopython. Constructing a pandas dataframe directly via dictionary.

Are you intending to use split method directly on the sequence, such that each residue is a separate column in a dataframe?

Anyway,

title_splits=re.findall(r"[\w']+", title) # Data cleaning is needed

This is a bad idea because you will remove the Genbank unique identifier (its an integer) and this should not leave the sequence.

You will need to switch out of the Fasta object. Your approacha and @user324810's use of

sequence = " ".join(sequence)

is okayish. However, I think it should be '', or "" (no space character). I spotted manually removing fasta, well the method should do this automatically. I am not familiar with Fastaparser, but AlignIO and SeqIO will both automatically process fasta without the need deal with ">". Biopython will have a length method and whilst you have the Biopython object it is preferable to use Biopython (OOP) methods.

from Bio.SeqIO.FastaIO import SimpleFastaParser 

This should be at the top of the code.

Overall, you manually open the data, pass it through the Bioparser then immediately dump the object. I do agree to write a pandas dataframe from a Biopython object these conversion to string, but Biopython will provide a method to print the object (pass it to a string) without all the parsing. Some of the parsing is removing key data (Genbank ID).

Personally I would look at the Stackoverflow example and base your code around this.

You clearly have an understanding of pandas, but I get the impression Biopython is something your not familiar with. My suspicion is that Biopython would infact answer the entirety of your question (whatever it might be) without passing to pandas.

pandas is cool and powerful for non-molecular data, but unless your using machine/deep learning (where a dataframe is necessary input) I think Biopython will provide most of the answers.

Important Terminology The class is FastaIO and the method is SimpleFastaParser. Humans and dogs are instances of this method and are therefore object(s).

Please don't write 'class' of dogs and humans in a formal report because it is incorrect OOP and will cause alarm if your reviewer is a coder. Your code is functional programming rather than OOP. You could say the human/dog fasta object, or just refer to it as human/dog dataframes and avoid any OOP terminology.

Answer 3

Just to add, usually the way to move a sequence object to a neat, decluttered string in Biopython is not to do this manually as done here:

title_splits=re.findall(r"[\w']+", title)
sequence = " ".join(sequence) 

both being from SimpleFastaParser(fastafile)

It should be possible and, if so, better to state,

 str(title)
 str(sequence)

If you can issue a print command and the string appears nice and neat on the screen str() will work.

This becomes important for complex sequence/alignment objects, such as SeqIO and AlignIO. The reason it works is __str__ has been declared or available through OOP inheritance.