Using a Bash Script to search TaxIDs against NCBI's Taxonomy yields "400 Bad Request" error?

Question

I've been searching TaxIDs against NCBI's Taxonomy DB to get taxonomic lineages for species. I have successfully done this for 1,000's of TaxIDs that were returned to me by Blast+ blasts in a CSV. (The first column in the CSV was the TaxID for the top hit of a sequence, and the other columns were for GI, Scientific Name, e-value, etc.). I simply wrote a script that would take the first field, which is always the TaxID, and search it against Taxonomy to get lineages for the species those TaxIDs represented. Everything worked fine!

However, some of the records returned from my original blast (that gave me the TaxIDs in the first place) would return two or more TaxIDs per record. (I suspect because a sequence matched equally well with two or more different species?) That's fine, but I had to remove these few exceptions before I searched the rest of the regular results against Taxonomy. So I've got most of my data/lineages, but for about 1:10,0000 records that gave multiple TaxIDs, I would like to know the lineages of the multiple species that the blast suggested.

I now took all of those "more-than-one-taxid" hits, separated them out into columns in an excel sheet and saved it as a CSV, and am trying to search these against Taxonomy with essentially the same script I used for the rest of the data, but am getting an error for every taxID I search (see below).

Complete Error (for each taxid searched against Taxonomy):

400 Bad Request
&rettype=full&retmode=xml&edirect_os=linux&edirect=7.90&tool=edirect&[email protected]'my&id=464126
Result of do_post http request is
$VAR1 = bless( {
                 '_protocol' => 'HTTP/1.1',
                 '_content' => '<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE eEfetchResult PUBLIC "-//NLM//DTD efetch 20131226//EN" "https://eutils.ncbi.nlm.nih.gov/eutils/dtd/20131226/efetch.dtd">
<eFetchResult>
	<ERROR>WebEnv parameter is required</ERROR>
</eFetchResult>
',
                 '_rc' => 400,
                 '_headers' => bless( {
                                        'connection' => 'close',
                                        'set-cookie' => 'ncbi_sid=9FF573148F458940_CEC6SID; domain=.nih.gov; path=/; expires=Thu, 14 Mar 2019 22:36:03 GMT',
                                        'cache-control' => 'private',
                                        'access-control-allow-origin' => '*',
                                        'date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'vary' => 'Accept-Encoding',
                                        'client-ssl-cert-issuer' => '/C=US/O=DigiCert Inc/OU=www.digicert.com/CN=DigiCert SHA2 High Assurance Server CA',
                                        'client-ssl-cipher' => 'ECDHE-RSA-AES256-GCM-SHA384',
                                        'client-peer' => '130.14.29.110:443',
                                        'strict-transport-security' => 'max-age=31536000; includeSubDomains; preload',
                                        'client-date' => 'Wed, 14 Mar 2018 22:36:03 GMT',
                                        'content-type' => 'text/xml; charset=UTF-8',
                                        'x-xss-protection' => '1; mode=block',
                                        'client-transfer-encoding' => [
                                                                        'chunked'
                                                                      ],
                                        'server' => 'Finatra',
                                        'client-ssl-socket-class' => 'IO::Socket::SSL',
                                        'client-response-num' => 1,
                                        'ncbi-phid' => 'DAF6EB0134DF5AEC039F11D248910BE7.1',
                                        'ncbi-sid' => '9FF573148F458940_CEC6SID',
                                        '::std_case' => {
                                                          'set-cookie' => 'Set-Cookie',
                                                          'access-control-allow-origin' => 'Access-Control-Allow-Origin',
                                                          'client-ssl-cert-issuer' => 'Client-SSL-Cert-Issuer',
                                                          'client-ssl-cipher' => 'Client-SSL-Cipher',
                                                          'client-peer' => 'Client-Peer',
                                                          'strict-transport-security' => 'Strict-Transport-Security',
                                                          'client-date' => 'Client-Date',
                                                          'x-xss-protection' => 'X-XSS-Protection',
                                                          'client-transfer-encoding' => 'Client-Transfer-Encoding',
                                                          'client-ssl-socket-class' => 'Client-SSL-Socket-Class',
                                                          'client-response-num' => 'Client-Response-Num',
                                                          'ncbi-phid' => 'NCBI-PHID',
                                                          'ncbi-sid' => 'NCBI-SID',
                                                          'x-ua-compatible' => 'X-UA-Compatible',
                                                          'client-ssl-cert-subject' => 'Client-SSL-Cert-Subject',
                                                          'content-security-policy' => 'Content-Security-Policy'
                                                        },
                                        'x-ua-compatible' => 'IE=Edge',
                                        'client-ssl-cert-subject' => '/C=US/ST=Maryland/L=Bethesda/O=National Library of Medicine/CN=*.ncbi.nlm.nih.gov',
                                        'content-security-policy' => 'upgrade-insecure-requests'
                                      }, 'HTTP::Headers' ),
                 '_msg' => 'Bad Request',
                 '_request' => bless( {
&rettype=full&retmode=xml&edirect_os=linux&edirect=7.90&tool=edirect&[email protected]',
                                        '_uri' => bless( do{\(my $o = 'https://eutils.ncbi.nlm.nih.gov/entrez/eutils/efetch.fcgi')}, 'URI::https' ),
                                        '_headers' => bless( {
                                                               'user-agent' => 'libwww-perl/6.05',
                                                               'content-type' => 'application/x-www-form-urlencoded',
                                                               '::std_case' => {
                                                                                 'if-ssl-cert-subject' => 'If-SSL-Cert-Subject'
                                                                               }
                                                             }, 'HTTP::Headers' ),
                                        '_method' => 'POST',
                                        '_uri_canonical' => $VAR1->{'_request'}{'_uri'}
                                      }, 'HTTP::Request' )
               }, 'HTTP::Response' );

What I've done:

Here is a sample of the data I used that worked correctly and yielded lineages when searching the taxIDs (taxIDs in 1st column):

549298,Contig1,Francisella halioticida,5.44e-126,gi|1213460577|gb|CP022132.1|,CP022132
240395,Contig2,Glandirana emeljanovi,4.74e-131,gi|1025812536|gb|KU641020.1|,KU641020
1069680,Contig3,Pneumocystis murina B123,2.34e-85,gi|1124120202|ref|XM_019757798.1|,XM_019757798
3871,Contig4,Lupinus angustifolius,1.89e-93,gi|1239725835|gb|CP023132.1|,CP023132

Here is a sample of the data that I am trying to get to work (when I search these manually against Taxonomy from their web page they seem to work and give lineages; it's just that I have just enough of them to make a non-programmatic solution really time consuming and tedious):

258234
1316279
1316315
710996
711000
225338
985895
710996
1382731
77133
344338

This is the business end of the script that I used that worked for the working data:

#!/bin/bash
while IFS=',' read -r f1 f2 f3 f4 f5 f6
do      
        #Firstly echo the Contig# (Query Seq. ID) followed by lineage information.
        echo -n "$f2," >> $2

        #Now get the lineage, pipe to xtract and parse XML for readability in output file
        efetch -db taxonomy \
        -id $f1 \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                ...
                -or Rank -equals species \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                >> $2
done < "$1"

This is the slightly altered script that I am trying to use to make the not-working set of data yield results (and that produces the error):

#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
        efetch -db taxonomy \
        -id $line \
        -format xml \
        | xtract -pattern Taxon \
                -tab ',' -sep ',' \
                -def "NONE" \
                -element TaxId,ScientificName \
                -division LineageEx \
                -group Taxon \
                -if Rank -equals superkingdom \
                -or Rank -equals kingdom \
                ...
                -or Rank -equals species \
                -or Rank -equals subspecies \
                -tab ',' -sep ',' \
                -def "NOT_AVAILABLE" \
                -element Rank,ScientificName \
                >> $2
done < "$1"

As you can see, the only difference is that I am reading in entire lines for the TaxIDs list because it is simply a list with line breaks between the IDs as opposed to a true CSV with other values as in the example that works. I've saved the taxIDs list as both a CSV and TXT and it doesn't make a difference. Also: I can echo what $line is in the second script and it is exactly I'd expect--just the numbers of the taxID, so why would I be getting a 400 error for bad syntax on my query if it looks exactly like what I am sending in the first, working script?

Thanks!

Answer 1

If I understand correctly, your question boils down to: parsing a bunch of numbers in a row from a .csv file gives a different result from parsing a bunch of numbers as rows of a text file. I that right? I suggest that you look at the source of your text file. For example, was the text file created on a Windows host (running bash, or in the command terminal?)?

Try using this modified version of the unix cat command: cat -et text_file.txt

The very end of each line will have a '\$' sign and that will show you if there are any extra non-visible characters in your rows. For example, a DOS/Windows formatted file will usually have a ^M at the end of each row before the '$'. If you don't strip off those line endings then they get added to your queries, even if you cannot see them. You can strip off those linefeed (or are they carriage returns?) using a Perl one-liner like: perl -pe 's/\r//' badfile.txt > goodfile.txt