I've been searching TaxIDs against NCBI's Taxonomy DB to get taxonomic lineages for species. I have successfully done this for 1,000's of TaxIDs that were returned to me by Blast+ blasts in a CSV. (The first column in the CSV was the TaxID for the top hit of a sequence, and the other columns were for GI, Scientific Name, e-value, etc.). I simply wrote a script that would take the first field, which is always the TaxID, and search it against Taxonomy to get lineages for the species those TaxIDs represented. Everything worked fine!

However, some of the records returned from my original blast (that gave me the TaxIDs in the first place) would return two or more TaxIDs per record. (I suspect because a sequence matched equally well with two or more different species?) That's fine, but I had to remove these few exceptions before I searched the rest of the regular results against Taxonomy. So I've got most of my data/lineages, but for about 1:10,0000 records that gave multiple TaxIDs, I would like to know the lineages of the multiple species that the blast suggested.

I now took all of those "more-than-one-taxid" hits, separated them out into columns in an excel sheet and saved it as a CSV, and am trying to search these against Taxonomy with essentially the same script I used for the rest of the data, but am getting an error for every taxID I search (see below).

Complete Error (for each taxid searched against Taxonomy):

400 Bad Request
Result of do_post http request is
$VAR1 = bless( {
                 '_protocol' => 'HTTP/1.1',
                 '_content' => '<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE eEfetchResult PUBLIC "-//NLM//DTD efetch 20131226//EN" "https://eutils.ncbi.nlm.nih.gov/eutils/dtd/20131226/efetch.dtd">
	<ERROR>WebEnv parameter is required</ERROR>
                 '_rc' => 400,
                 '_headers' => bless( {
                                        'connection' => 'close',
                                        'set-cookie' => 'ncbi_sid=9FF573148F458940_CEC6SID; domain=.nih.gov; path=/; expires=Thu, 14 Mar 2019 22:36:03 GMT',
                                        'cache-control' => 'private',
                                        'access-control-allow-origin' => '*',
                                        'date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'vary' => 'Accept-Encoding',
                                        'client-ssl-cert-issuer' => '/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA',
                                        'client-ssl-cipher' => 'ECDHE-RSA-AES256-GCM-SHA384',
                                        'client-peer' => '',
                                        'strict-transport-security' => 'max-age=31536000; includeSubDomains; preload',
                                        'client-date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'content-type' => 'text/xml; charset=UTF-8',
                                        'x-xss-protection' => '1; mode=block',
                                        'client-transfer-encoding' => [
                                        'server' => 'Finatra',
                                        'client-ssl-socket-class' => 'IO::Socket::SSL',
                                        'client-response-num' => 1,
                                        'ncbi-phid' => 'DAF6EB0134DF5AEC039F11D248910BE7.1',
                                        'ncbi-sid' => '9FF573148F458940_CEC6SID',
                                        '::std_case' => {
                                                          'set-cookie' => 'Set-Cookie',
                                                          'access-control-allow-origin' => 'Access-Control-Allow-Origin',
                                                          'client-ssl-cert-issuer' => 'Client-SSL-Cert-Issuer',
                                                          'client-ssl-cipher' => 'Client-SSL-Cipher',
                                                          'client-peer' => 'Client-Peer',
                                                          'strict-transport-security' => 'Strict-Transport-Security',
                                                          'client-date' => 'Client-Date',
                                                          'x-xss-protection' => 'X-XSS-Protection',
                                                          'client-transfer-encoding' => 'Client-Transfer-Encoding',
                                                          'client-ssl-socket-class' => 'Client-SSL-Socket-Class',
                                                          'client-response-num' => 'Client-Response-Num',
                                                          'ncbi-phid' => 'NCBI-PHID',
                                                          'ncbi-sid' => 'NCBI-SID',
                                                          'x-ua-compatible' => 'X-UA-Compatible',
                                                          'client-ssl-cert-subject' => 'Client-SSL-Cert-Subject',
                                                          'content-security-policy' => 'Content-Security-Policy'
                                        'x-ua-compatible' => 'IE=Edge',
                                        'client-ssl-cert-subject' => '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/CN=*.ncbi.nlm.nih.gov',
                                        'content-security-policy' => 'upgrade-insecure-requests'
                                      }, 'HTTP::Headers' ),
                 '_msg' => 'Bad Request',
                 '_request' => bless( {
                                        '_uri' => bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi')}, 'URI::https' ),
                                        '_headers' => bless( {
                                                               'user-agent' => 'libwww-perl/6.05',
                                                               'content-type' => 'application/x-www-form-urlencoded',
                                                               '::std_case' => {
                                                                                 'if-ssl-cert-subject' => 'If-SSL-Cert-Subject'
                                                             }, 'HTTP::Headers' ),
                                        '_method' => 'POST',
                                        '_uri_canonical' => $VAR1->{'_request'}{'_uri'}
                                      }, 'HTTP::Request' )
               }, 'HTTP::Response' );

What I've done:

Here is a sample of the data I used that worked correctly and yielded lineages when searching the taxIDs (taxIDs in 1st column):

549298,Contig1,Francisella halioticida,5.44e-126,gi|1213460577|gb|CP022132.1|,CP022132
240395,Contig2,Glandirana emeljanovi,4.74e-131,gi|1025812536|gb|KU641020.1|,KU641020
1069680,Contig3,Pneumocystis murina B123,2.34e-85,gi|1124120202|ref|XM_019757798.1|,XM_019757798
3871,Contig4,Lupinus angustifolius,1.89e-93,gi|1239725835|gb|CP023132.1|,CP023132

Here is a sample of the data that I am trying to get to work (when I search these manually against Taxonomy from their web page they seem to work and give lineages; it's just that I have just enough of them to make a non-programmatic solution really time consuming and tedious):


This is the business end of the script that I used that worked for the working data:

while IFS=',' read -r f1 f2 f3 f4 f5 f6
        #Firstly echo the Contig# (Query Seq. ID) followed by lineage information.
        echo -n "$f2," >> $2

        #Now get the lineage, pipe to xtract and parse XML for readability in output file
        efetch -db taxonomy \
        -id $f1 \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                -or Rank -equals species \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                >> $2
done < "$1"

This is the slightly altered script that I am trying to use to make the not-working set of data yield results (and that produces the error):

while IFS='' read -r line || [[ -n "$line" ]]; do
        efetch -db taxonomy \
        -id $line \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                -or Rank -equals species \
                -or Rank -equals subspecies \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                >> $2
done < "$1"

As you can see, the only difference is that I am reading in entire lines for the TaxIDs list because it is simply a list with line breaks between the IDs as opposed to a true CSV with other values as in the example that works. I've saved the taxIDs list as both a CSV and TXT and it doesn't make a difference. Also: I can echo what $line is in the second script and it is exactly I'd expect--just the numbers of the taxID, so why would I be getting a 400 error for bad syntax on my query if it looks exactly like what I am sending in the first, working script?


  • 1
    $\begingroup$ Hello, nice question. Just, could you please tame (edit) a bit the error log? It's very hard to look look at. $\endgroup$
    – Kamil S Jaron
    Mar 15 '18 at 14:13
  • $\begingroup$ Aside from your specific question slightly, you might be interested in looking at the NCBITaxa() tool in the ETE3 package. It basically does what your script does, but provides lots of other useful methods for querying the ncbi taxa database. $\endgroup$
    – Joe Healey
    Jul 30 '18 at 20:51

If I understand correctly, your question boils down to: parsing a bunch of numbers in a row from a .csv file gives a different result from parsing a bunch of numbers as rows of a text file. I that right? I suggest that you look at the source of your text file. For example, was the text file created on a Windows host (running bash, or in the command terminal?)?

Try using this modified version of the unix cat command: cat -et text_file.txt

The very end of each line will have a '\$' sign and that will show you if there are any extra non-visible characters in your rows. For example, a DOS/Windows formatted file will usually have a ^M at the end of each row before the '$'. If you don't strip off those line endings then they get added to your queries, even if you cannot see them. You can strip off those linefeed (or are they carriage returns?) using a Perl one-liner like: perl -pe 's/\r//' badfile.txt > goodfile.txt

  • $\begingroup$ Thanks, I suspect that between generating these files on my school's servers through bash to downloading them locally and manipulating them in Excel, that there must've been '$' characters like you said. Since then, I've learned to do what I was doing in Excel on the command line and think I'll be able to avoid this altogether. $\endgroup$
    – ljs
    Aug 2 '18 at 0:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.