2
$\begingroup$

I have metabarcoding sequence data (COI) from bulk animal samples (including arthropoda, nematoda, annelida, mollusca) and I want to BLAST all of these sequences. I used following command to do this: blastn -remote -db nt -query COI_all.fasta -num_alignments 2 -out COI_blasted.txt. However this results in errors similar to this post: https://www.biostars.org/p/359971/ .

These errors probably appear due to the number of sequences in my file (around 700) and the remote connection is thus interrupted.

I found that a solution would be to use blastn with a local database and since the samples are so diverse, I would like to download ALL animal COI sequences from BOLD (or gen bank). It would not be a problem if non-animal (e.g. plant) sequences would also be included.

I think the BOLD database would be great to BLAST my sequences to. However, I'm currently struggling to find a good way to download all animal COI sequences from BOLD.

When entering COI-5P as search term on http://v4.boldsystems.org/index.php/Public_SearchTerms I receive error: Your search terms resulted in too many matching terms. Please try again with more specific search criteria.. I could likely download the sequences from all the phyla etc seperately and merge them, but I'd rather just download 1 file.

I also tried to use the API by running: wget http://v4.boldsystems.org/index.php/API_Public/sequence?marker=COI-5P. A download starts but around 3.7 MB download, it is stuck and the file I receive only contains ~5000 sequences.

UPDATE: I've contacted BOLD about the stalling behavior and this is their reply: "This issue is because of the large API request that retrieves millions of records, which our system does not handle. Please break up the search by smaller groups, such as classes."

Does anyone have a solution to download all COI sequences from BOLD in one file?

I could also download COI sequences from gen bank using the ftp://ftp.ncbi.nlm.nih.gov/blast/db/ URL, but I'm not sure which exact files I need. For 16S, 18S,.. it is obvious, but not for COI. Any suggestions?

Thanks for the help.

$\endgroup$
1
+50
$\begingroup$

A straightforward way of getting the sequences from GenBank is from their website using this URL: https://www.ncbi.nlm.nih.gov/nuccore?term=COI[Gene Name]

There you can click on "Send to" -> select "Gene Features" and "FASTA Nucleotide" -> Click "Create File".

There are >3 mio hits, so the download could take a while.

$\endgroup$
7
  • $\begingroup$ Thank for your suggestion, this is certainly a possibility for the Gen Bank data. For now, I see Gen Bank data as a good alternative, but am still looking for a solution to the BOLD database. $\endgroup$
    – Robvh
    May 4 '20 at 9:41
  • 1
    $\begingroup$ I tried to download the sequences from BOLD with wget using the API link you provided above, and got the same stalling behaviour that you described. Maybe they apply some sort of limit per download. I'm not really familliar with BOLD. $\endgroup$
    – mrhd
    May 4 '20 at 11:17
  • $\begingroup$ Thanks for replicating my problem, then I know my computer is not the problem. I will contact them and ask for any limitations. I will post back their reply. $\endgroup$
    – Robvh
    May 4 '20 at 11:29
  • $\begingroup$ downloading all COI sequences with the method you provide in this answer also does not work. Around ~150MB download the downloading stops with error "failed-network error". I assume this is also some download limitation as we do not experience any network errors? $\endgroup$
    – Robvh
    May 6 '20 at 6:25
  • 1
    $\begingroup$ I'm now downloading all COI gen bank sequences withesearch from the E-Utilities package in the Ubuntu terminal using command: esearch -db nucleotide -query "COI[GENE]" | efetch -format fasta > COI_all_genbank.fasta. This should do the trick. $\endgroup$
    – Robvh
    May 6 '20 at 7:35
1
$\begingroup$

I couldn't help but looking a bit deeper into this. There is actually an R package that uses the BOLD API.

However, they also say that there will be timeouts when downloading large sets of data. Luckily, they suggest a workaround (downloading sequences separately for each taxonomic Class).

So you should be able to download the marker sequences using the suggested steps:

# install packages (if necessary)
install.packages("bold")
install.packages("taxize")
install.packages("seqinr")
# load packages
library(tidyverse)
library(bold)    # API interface to BOLD
library(taxize)  # for NCBI taxonomy lookup
library(seqinr)  # for FASTA output

# get class-level taxa within "Animalia" from NCBI taxonomy
taxa <- downstream("Animalia", db = "ncbi", downto = "class")
# check if taxa present in BOLD
checks <- bold_tax_name(taxa$Animalia$childtaxa_name)
taxa_bold <- checks[!is.na(checks$taxon),]$taxon

# download sequences from BOLD for each class-level taxon
sequences <- map(taxa_bold[1:3], bold_seq, marker = 'COI-5P') %>%
  flatten() %>%
  bind_rows()

# write sequences to file
write.fasta(
  sequences = as.list(sequences$sequence), 
  names = as.list(sequences$id), 
  nbchar = 80, 
  file.out = 'coi5p.fasta')

Note that I selected the first tree taxa (taxa_bold[1:3]) for testing, so that it runs quicker. If you want to get all taxa, just remove the [1:3].

One last importand caveat: In the help for bold_seq it says that

Notes from BOLD on the marker param: "All markers for a specimen matching the search stringwill be returned. ie. A record with COI-5P and ITS will return sequence data for both markers evenif only COI-5P was specified."You will likely end up with data with markers that you did not request - just be sure to filter thoseout as needed.

So you might want to double-check that the downloaded sequences are actually COI-5P. But if you are using them as a BLAST database, you might also be happy with COI-5P and ITS markers being present in the database (it affects the e-values, though).

$\endgroup$
4
  • 1
    $\begingroup$ I tried it and when downloading all sequences I receive following warnings: Warning messages: 1: In .f(.x[[i]], ...) : the request timed out, see 'If a request times out' returning partial output 2: In .f(.x[[i]], ...) : the request timed out, see 'If a request times out' returning partial output 3: In .f(.x[[i]], ...) : the request timed out, see 'If a request times out' returning partial output it downloaded 1012676 sequences, so quiet alot but I don't think that all are downloaded. Again, these warnings seem to be related to some kind of download limitation.. $\endgroup$
    – Robvh
    May 6 '20 at 6:45
  • 1
    $\begingroup$ If there are timeouts one must assume that not all sequences have been downloaded. You could also try going a to a deeper taxonomic rank than class (order?) in case that some classes have so many sequences that it leads to a timeout. $\endgroup$
    – mrhd
    May 6 '20 at 7:55
  • $\begingroup$ I think I will use the esearch as final tool, as it is more to the point than the BOLD downloads. Thanks for your extensive help regarding the issue, for which I awarded the bounty. $\endgroup$
    – Robvh
    May 6 '20 at 8:37
  • 1
    $\begingroup$ Thank you @Robvh! If I had to choose for myself I would also go with GenBank as a source. $\endgroup$
    – mrhd
    May 6 '20 at 9:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.