4
$\begingroup$

I have metabarcoding sequence data (COI) from bulk animal samples (including arthropoda, nematoda, annelida, mollusca) and I want to BLAST all of these sequences. I used following command to do this: blastn -remote -db nt -query COI_all.fasta -num_alignments 2 -out COI_blasted.txt. However this results in errors similar to this post: https://www.biostars.org/p/359971/ .

These errors probably appear due to the number of sequences in my file (around 700) and the remote connection is thus interrupted.

I found that a solution would be to use blastn with a local database and since the samples are so diverse, I would like to download ALL animal COI sequences from BOLD (or gen bank). It would not be a problem if non-animal (e.g. plant) sequences would also be included.

I think the BOLD database would be great to BLAST my sequences to. However, I'm currently struggling to find a good way to download all animal COI sequences from BOLD.

When entering COI-5P as search term on http://v4.boldsystems.org/index.php/Public_SearchTerms I receive error: Your search terms resulted in too many matching terms. Please try again with more specific search criteria.. I could likely download the sequences from all the phyla etc seperately and merge them, but I'd rather just download 1 file.

I also tried to use the API by running: wget http://v4.boldsystems.org/index.php/API_Public/sequence?marker=COI-5P. A download starts but around 3.7 MB download, it is stuck and the file I receive only contains ~5000 sequences.

UPDATE: I've contacted BOLD about the stalling behavior and this is their reply: "This issue is because of the large API request that retrieves millions of records, which our system does not handle. Please break up the search by smaller groups, such as classes."

Does anyone have a solution to download all COI sequences from BOLD in one file?

I could also download COI sequences from gen bank using the ftp://ftp.ncbi.nlm.nih.gov/blast/db/ URL, but I'm not sure which exact files I need. For 16S, 18S,.. it is obvious, but not for COI. Any suggestions?

Thanks for the help.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

I couldn't help but looking a bit deeper into this. There is actually an R package that uses the BOLD API.

However, they also say that there will be timeouts when downloading large sets of data. Luckily, they suggest a workaround (downloading sequences separately for each taxonomic Class).

So you should be able to download the marker sequences using the suggested steps:

# install packages (if necessary)
install.packages("bold")
install.packages("taxize")
install.packages("seqinr")
# load packages
library(tidyverse)
library(bold)    # API interface to BOLD
library(taxize)  # for NCBI taxonomy lookup
library(seqinr)  # for FASTA output

# get class-level taxa within "Animalia" from NCBI taxonomy
taxa <- downstream("Animalia", db = "ncbi", downto = "class")
# check if taxa present in BOLD
checks <- bold_tax_name(taxa$Animalia$childtaxa_name)
taxa_bold <- checks[!is.na(checks$taxon),]$taxon

# download sequences from BOLD for each class-level taxon
sequences <- map(taxa_bold[1:3], bold_seq, marker = 'COI-5P') %>%
  flatten() %>%
  bind_rows()

# write sequences to file
write.fasta(
  sequences = as.list(sequences$sequence), 
  names = as.list(sequences$id), 
  nbchar = 80, 
  file.out = 'coi5p.fasta')

Note that I selected the first tree taxa (taxa_bold[1:3]) for testing, so that it runs quicker. If you want to get all taxa, just remove the [1:3].

One last importand caveat: In the help for bold_seq it says that

Notes from BOLD on the marker param: "All markers for a specimen matching the search stringwill be returned. ie. A record with COI-5P and ITS will return sequence data for both markers evenif only COI-5P was specified."You will likely end up with data with markers that you did not request - just be sure to filter thoseout as needed.

So you might want to double-check that the downloaded sequences are actually COI-5P. But if you are using them as a BLAST database, you might also be happy with COI-5P and ITS markers being present in the database (it affects the e-values, though).

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ I tried it and when downloading all sequences I receive following warnings: Warning messages: 1: In .f(.x[[i]], ...) : the request timed out, see 'If a request times out' returning partial output 2: In .f(.x[[i]], ...) : the request timed out, see 'If a request times out' returning partial output 3: In .f(.x[[i]], ...) : the request timed out, see 'If a request times out' returning partial output it downloaded 1012676 sequences, so quiet alot but I don't think that all are downloaded. Again, these warnings seem to be related to some kind of download limitation.. $\endgroup$
    – RobH
    Commented May 6, 2020 at 6:45
  • 1
    $\begingroup$ If there are timeouts one must assume that not all sequences have been downloaded. You could also try going a to a deeper taxonomic rank than class (order?) in case that some classes have so many sequences that it leads to a timeout. $\endgroup$
    – mrhd
    Commented May 6, 2020 at 7:55
  • $\begingroup$ I think I will use the esearch as final tool, as it is more to the point than the BOLD downloads. Thanks for your extensive help regarding the issue, for which I awarded the bounty. $\endgroup$
    – RobH
    Commented May 6, 2020 at 8:37
  • 1
    $\begingroup$ Thank you @Robvh! If I had to choose for myself I would also go with GenBank as a source. $\endgroup$
    – mrhd
    Commented May 6, 2020 at 9:06
  • 1
    $\begingroup$ @mrhd hey so I took interest in this script. The bind_rows() command returns the following error. Error in bind_rows(): ! Argument 1 must be a data frame or a named atomic vector. Run rlang::last_trace() to see where the error occurred. Can you lend a helping hand after all these years? $\endgroup$
    – Nickmofoe
    Commented Oct 28, 2023 at 9:01
1
+50
$\begingroup$

A straightforward way of getting the sequences from GenBank is from their website using this URL: https://www.ncbi.nlm.nih.gov/nuccore?term=COI[Gene Name]

There you can click on "Send to" -> select "Gene Features" and "FASTA Nucleotide" -> Click "Create File".

There are >3 mio hits, so the download could take a while.

Improve this answer
$\endgroup$
7
  • $\begingroup$ Thank for your suggestion, this is certainly a possibility for the Gen Bank data. For now, I see Gen Bank data as a good alternative, but am still looking for a solution to the BOLD database. $\endgroup$
    – RobH
    Commented May 4, 2020 at 9:41
  • 1
    $\begingroup$ I tried to download the sequences from BOLD with wget using the API link you provided above, and got the same stalling behaviour that you described. Maybe they apply some sort of limit per download. I'm not really familliar with BOLD. $\endgroup$
    – mrhd
    Commented May 4, 2020 at 11:17
  • $\begingroup$ Thanks for replicating my problem, then I know my computer is not the problem. I will contact them and ask for any limitations. I will post back their reply. $\endgroup$
    – RobH
    Commented May 4, 2020 at 11:29
  • $\begingroup$ downloading all COI sequences with the method you provide in this answer also does not work. Around ~150MB download the downloading stops with error "failed-network error". I assume this is also some download limitation as we do not experience any network errors? $\endgroup$
    – RobH
    Commented May 6, 2020 at 6:25
  • 1
    $\begingroup$ I'm now downloading all COI gen bank sequences withesearch from the E-Utilities package in the Ubuntu terminal using command: esearch -db nucleotide -query "COI[GENE]" | efetch -format fasta > COI_all_genbank.fasta. This should do the trick. $\endgroup$
    – RobH
    Commented May 6, 2020 at 7:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.