0
$\begingroup$

I have 193000 protein interactions in CSV named all_proteininteractions.csv which have query protein name in the 2nd column and partner protein in the 3rd like this:

Query_ENSP,Query_Name,Partner_Name,Combined_Score
9606.ENSP00000000233,ARF5,RAC2,0.4
9606.ENSP00000000233,ARF5,RAC2,0.4
9606.ENSP00000000233,ARF5,PPP2CB,0.408
9606.ENSP00000000233,ARF5,PPP2CB,0.408
9606.ENSP00000000233,ARF5,R3HDML,0.413
9606.ENSP00000000233,ARF5,R3HDML,0.413
9606.ENSP00000000233,ARF5,PLEK,0.422
9606.ENSP00000000233,ARF5,PLEK,0.422
9606.ENSP00000000233,ARF5,RAB11FIP5,0.501
9606.ENSP00000000233,ARF5,RAB11FIP5,0.501
9606.ENSP00000000233,ARF5,AP3B1,0.553
9606.ENSP00000000233,ARF5,AP3B1,0.553
9606.ENSP00000000233,ARF5,CYTH4,0.579
9606.ENSP00000000233,ARF5,CYTH4,0.579
9606.ENSP00000000233,ARF5,RABAC1,0.658
9606.ENSP00000000233,ARF5,RABAC1,0.658
9606.ENSP00000000233,ARF5,YKT6,0.672
9606.ENSP00000000233,ARF5,YKT6,0.672
9606.ENSP00000000233,ARF5,BET1,0.679

I am using this Python code to fetch sequences of each protein and store them in a CSV file:

import pandas as pd
import requests
import math

requests.packages.urllib3.disable_warnings()

# Function to fetch protein sequence from UniProt
def fetch_protein_sequence_from_uniprot(protein_name):
    #Declaring UniProt API
    uniport_api_url = f"https://rest.uniprot.org/uniprotkb/search?query=gene_exact:{protein_name}+AND+organism_id:9606&format=fasta&size=1"
    response = requests.get(uniport_api_url,verify=False)

    # Parse the response to extract sequence
    sequence = ""
    if response.ok:
        lines = response.text.split("\n")
        for line in lines:
            if not line.startswith(">"):  # Exclude header lines
                sequence += line
    return sequence

# Reading the input CSV file
input_csv = "all_protein_interactions.csv"
chunksize = 1000  # Batch size

# Create an iterator to read the CSV in chunks
csv_reader = pd.read_csv(input_csv, chunksize=chunksize)

# Initialize an empty list to store modified chunks
modified_chunks = []

# Process each chunk
for chunk in csv_reader:
    # Fetch protein sequences for query and partner names
    query_sequences = [fetch_protein_sequence_from_uniprot(name) for name in chunk.iloc[:, 1]]
    partner_sequences = [fetch_protein_sequence_from_uniprot(name) for name in chunk.iloc[:, 2]]

    # Add fetched sequences as new columns in the chunk
    chunk['Query_Sequence'] = query_sequences
    chunk['Partner_Sequence'] = partner_sequences

    # Append the modified chunk to the list
    modified_chunks.append(chunk)

# Concatenate all modified chunks into a single DataFrame
result_df = pd.concat(modified_chunks, ignore_index=True)

# Save the updated DataFrame back to CSV
result_df.to_csv("protein_interactionswithseqs.csv", index=False)

First, it was throwing SSLError, so I made verify=False. But now, after running the script for 3 hours it's showing following error:

ConnectionError: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response'))

How can I remove this connection aborted error to fetch 386000 sequences?

$\endgroup$
3
  • $\begingroup$ You also should add print statements in various places, to see where the problem is. Has the script managed to finish the for chunk loop? Where exactly is it failing? And isn't your loop trying to get the same protein multiple times? You seem to have the same name in the 2nd column for several lines, so why get that same protein multiple times? $\endgroup$
    – terdon
    Jan 7 at 12:45
  • $\begingroup$ Thanks. I am using Windows 10 with Intel(R) Core(TM) i7-3770 CPU and 12 GB ram. $\endgroup$ Jan 7 at 12:46
  • $\begingroup$ I modified the text as well. I have similar proteins but I want to fetch them all and store in 5th and 6th column $\endgroup$ Jan 7 at 12:53

1 Answer 1

1
$\begingroup$

Your script works as expected on the example you gave. Running your script results in a protein_interactionswithseqs.csv with lines like:

$ head -n2 protein_interactionswithseqs.csv 
Query_ENSP,Query_Name,Partner_Name,Combined_Score,Query_Sequence,Partner_Sequence
9606.ENSP00000000233,ARF5,RAC2,0.4,MGLTVSALFSRIFGKKQMRILMVGLDAAGKTTILYKLKLGEIVTTIPTIGFNVETVEYKNICFTVWDVGGQDKIRPLWRHYFQNTQGLIFVVDSNDRERVQESADELQKMLQEDELRDAVLLVFANKQDMPNAMPVSELTDKLGLQHLRSRTWYVQATCATQGTGLYDGLDWLSHELSKR,MQAIKCVVVGDGAVGKTCLLISYTTNAFPGEYIPTVPGRPCALLQDAEAGLSSDVEGACWCQTQEGRMTGEAPAPPWPALRGPHHS

However, this is horribly, horribly inefficient! You are fetching the same sequence multiple times. With your example data, the protein ARF5 is actually downloaded 19 times, meaning this will take 18 times more time than needed. This is very likely the cause for the timeout error you describe. Now, it might be that your sequences are too many to be stored in RAM (but I would still try it), so one way around this is to download the sequences first, and then write the mappings. Something like this:

import pandas as pd
import requests
import sys
requests.packages.urllib3.disable_warnings()


def write_protein_seq_file(protein_name, protein_seq):
    file_name = "seqs/" + protein_name
    seq_file = open(file_name, "w")
    data = f"{protein_seq}\n"
    seq_file.write(data)
    seq_file.close

def read_protein_seq_file(protein_name):
    seq_file = open("seqs/" + protein_name, "r")
    data = seq_file.read()
    seq_file.close
    return data.rstrip()

# Function to fetch protein sequence from UniProt
def fetch_protein_sequence_from_uniprot(protein_name):
    #Declaring UniProt API
    uniport_api_url = f"https://rest.uniprot.org/uniprotkb/search?query=gene_exact:{protein_name}+AND+organism_id:9606&format=fasta&size=1"
    response = requests.get(uniport_api_url,verify=False)

    # Parse the response to extract sequence
    sequence = ""
    if response.ok:
        lines = response.text.split("\n")
        for line in lines:
            if not line.startswith(">"):  # Exclude header lines
                sequence += line
    return sequence

# Reading the input CSV file
input_csv = "all_protein_interactions.csv"
chunksize = 1000  # Batch size

# Create an iterator to read the CSV in chunks
csv_reader = pd.read_csv(input_csv, chunksize=chunksize)

# Initialize an empty list to store modified chunks
modified_chunks = []

# Get all sequences
for chunk in csv_reader:
    for protein in set(list(chunk.iloc[:, 1]) + list(chunk.iloc[:, 2])):
        protein_seq = fetch_protein_sequence_from_uniprot(protein) 
        write_protein_seq_file(protein,protein_seq)

print("Finished fetching sequences", file=sys.stderr)     
## Now, process the file again and write the new file
csv_reader = pd.read_csv(input_csv, chunksize=chunksize)

for chunk in csv_reader:

    # Fetch protein sequences for query and partner names
    query_sequences = [read_protein_seq_file(name) for name in chunk.iloc[:, 1]]
    partner_sequences = [read_protein_seq_file(name) for name in chunk.iloc[:, 2]]

    # Add fetched sequences as new columns in the chunk
    chunk['Query_Sequence'] = query_sequences
    chunk['Partner_Sequence'] = partner_sequences

    # Append the modified chunk to the list
    modified_chunks.append(chunk)

# Concatenate all modified chunks into a single DataFrame
result_df = pd.concat(modified_chunks, ignore_index=True)

# Save the updated DataFrame back to CSV
result_df.to_csv("protein_interactionswithseqs.csv", index=False) 

First, create a directory named seqs, then run the script above. It will process the csv file twice. Once to get all sequences mentioned in the file and save each one in a file called seqs/$sequence_name (where $sequence_name is replaced by the actual name of each protein. Then, it goes through the file a second time, but instead of the very time consuming download, it now fetches the sequence from the file stored on your computer. Once you are satisfied this works, you can delete the seqs directory.

Note that I don't have access to a Windows machine, so this was tested on Linux. Also, I am not a python coder, so I am sure this ins't very good python and there will be improvements to be made, but even on the toy example you gave, this speeds things up significantly:

$ time your_script.py 

real    0m10.186s
user    0m0.998s
sys     0m0.076s

$ time my_script.py
Finished fetching sequences

real    0m3.180s
user    0m0.456s
sys     0m0.066s

As you can see above, even on the tiny example with only 19 data lines, your script took 10 seconds while mine only took 3. So try this approach (or, better, take my example and make it more pythonic, add some error checking, some try/except blocks, make sure your data are correct) and you might find the timeout error is no longer an issue. You should certainly find that the whole thing takes far less than three hours(!) to run.

First of all, that is a really, really inefficient way of doing it. Using your example data, you are getting the same protein, ARF5, 19 times. This makes your program 18 times slower than it needs to be. You should modify the script to download each protein once and then write them all out.

$\endgroup$
2
  • $\begingroup$ Your example script is not taking any file and where to download the all sequences $\endgroup$ Jan 7 at 14:23
  • $\begingroup$ @AnasJamshed what do you mean? It is taking the exact same file as your version and in the exact same way. The sequences will be downloaded into the seqs directory which, as I explain in my answer, you need to create. $\endgroup$
    – terdon
    Jan 7 at 16:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.