Shell script to validate fastq issue

Question

I have two GSM ID as test case where both of them are having data when I checked through sra explorer tool but when I try to fetch through a script only one of them returns output where the other one fails.

GSM ID GSM3603268 GSM2683458

Output For GSM3603268

$ cat GSM3603268.test.fastq
@GSM3603268.1 1 length=100
TTTTTTAAAGAAAACTTGAGCTTTTGGAGCATGGCAACCTAGCCTGCAGACACCGTATCCCCTTGTCCACTTCCCCCTGCAAACCATAAGTCCATTCCTA
+GSM3603268.1 1 length=100
=@@DDDDA++2CDABCEG@GEHHEHGA9EDFHGHD3::B??<DBFFBFB===)5;CA;(7=?;?CE;7)7;>C((583;<9>(,89@@@@CCCCDD:BC:
@GSM3603268.1 1 length=100
AGTTCCACGAGTTTTCTTTTTTTTAAGTGGTAGGAATGGACTTATGGGTTGCAGGGGGAAGTGGACAAGGGGCTACGGTGTCTGCAGGCTTGGTTGCCAT
+GSM3603268.1 1 length=100
:?=BD:D?C)<CDGH4+<CCCGHI)??D<D?00?D*?39C<CG4BFE28;4(.7AB############################################

$ cat GSM3603268.test.log
2023-09-26T21:18:46 fastq-dump.3.0.1 int: string unexpected while executing query within virtual file system module - multiple response SRR URLs for the same service 's3'
2023-09-26T21:18:46 fastq-dump.3.0.1 int: string unexpected while executing query within virtual file system module - multiple response SRR URLs for the same service 's3'
2023-09-26T21:18:46 fastq-dump.3.0.1 int: string unexpected while executing query within virtual file system module - multiple response SRR URLs for the same service 's3'
Read 1 spots for GSM3603268
Written 1 spots for GSM3603268

Output for GSM2683458

$ cat GSM2683458.test.fastq
    bash-5.1#

$ cat GSM2683458.test.log
2023-09-26T21:16:55 fastq-dump.3.0.1 int: string unexpected while executing query within virtual file system module - multiple response SRR URLs for the same service 's3'
2023-09-26T21:16:55 fastq-dump.3.0.1 int: string unexpected while executing query within virtual file system module - multiple response SRR URLs for the same service 's3'
2023-09-26T21:16:55 fastq-dump.3.0.1 int: string unexpected while executing query within virtual file system module - multiple response SRR URLs for the same service 's3'
2023-09-26T21:17:11 fastq-dump.3.0.1 warn: directory not found while opening manager within virtual file system module - can't open NC_000076.5 as a RefSeq or as a WGS
2023-09-26T21:17:11 fastq-dump.3.0.1 err: directory not found while opening manager within virtual file system module - failed GSM2683458

=============================================================
An error occurred during processing.
A report was generated into the file '/root/ncbi_error_report.txt'.
If the problem persists, you may consider sending the file
to '[email protected]' for assistance.
=============================================================

fastq-dump quit with error code 3

The script

#!/bin/sh

set -x
PS4='[\\d \\t] '

# Check parameter for error
check=0
# Print fastq-dump executable path
echo $(which fastq-dump)

# Function to download FastQ with retries
download_with_retry() {
    local id="$1"
local retries=5
for attempt in $(seq "$retries"); do
    # Download start of fastq
    fastq-dump $(get_ngc) -X 1 -Z --split-spot "$id" > "${id}.test.fastq" 2> "${id}.test.log"
    numLines=$(cat "${id}.test.fastq" | wc -l)
    if [ $numLines -gt 0 ]; then
            echo "${id} has data... OK"
        return 0
    else
        echo "${id} does not have data on attempt $attempt... Retrying in 5 seconds..."
        sleep 5
    fi
done
echo "${id} could not be downloaded after $retries attempts... ERROR"
    check=1
    return 1
}

# Loop through all parameters to check validity
for file in "$@"; do
cp "${file}" .
    # Extract filename for sampleID
    file_basename=$(basename "${file}")
    id="${file_basename%".id"}"
# Start validation with retries
echo "Checking ${id}..."
    download_with_retry "$id"
done

# Exit with error if some FastQs are not accessible
if [ $check -gt 0 ]; then
    echo "ERROR: One or more samples have inaccessible FastQs.. exiting"
    exit 1
fi

What exactly is going wrong in the case of the other ID? No matter how many retries I give, it fails.

Any suggestion or help would be really appreciated.

The error log:

$ cat ncbi_error_report.txt
<Report>
 <Run>
  <Date>
   <Start value="Tue Sep 26 2023 9:16:55 PM"/>
   <End value="Tue Sep 26 2023 9:17:11 PM"/>
  </Date>
  <Home name="HOME" value="/root"/>
  <Cwd>/tmp</Cwd>
  <CommandLine argc="6">
   <Arg index="0" value="fastq-dump"/>
   <Arg index="1" value="-X"/>
   <Arg index="2" value="1"/>
   <Arg index="3" value="-Z"/>
   <Arg index="4" value="--split-spot"/>
   <Arg index="5" value="GSM2683458"/>
  </CommandLine>
  <Result rc="RC(rcVFS,rcMgr,rcOpening,rcDirectory,rcNotFound)"/>
  <User admin="true"/>
 </Run>
 <Configuration>
  <Files count="2">
   <File name="/etc/ncbi/settings.kfg"/>
   <File name="/root/.ncbi/user-settings.mkfg"/>
  </Files>
  <refseq state="not found"/>
  <krypto state="pwfile: not found"/>
  <sra>
   <quality_type>raw_scores</quality_type>
  </sra>
  <Config>
  <ConfigurationFiles>
/etc/ncbi/settings.kfg
/root/.ncbi/user-settings.mkfg
    </ConfigurationFiles>
    <APPNAME>"fastq-dump"</APPNAME>
    <APPPATH>"/tmp/"</APPPATH>
    <BUILD>"RELEASE"</BUILD>
    <HOME>"/root"</HOME>
    <HOST></HOST>
    <LIBS>
      <GUID>"119c217a-7b81-47e8-91d6-56d19c8c9f15"</GUID>
      <IMAGE_GUID>"119c217a-7b81-47e8-91d6-62229c64ee59"</IMAGE_GUID>
    </LIBS>
    <NCBI_HOME>"/root/.ncbi"</NCBI_HOME>
    <NCBI_SETTINGS>"/root/.ncbi/user-settings.mkfg"</NCBI_SETTINGS>
    <OS>"linux"</OS>
    <PWD>"/tmp"</PWD>
    <USER></USER>
    <VDB_CONFIG></VDB_CONFIG>
    <VDB_ROOT></VDB_ROOT>
    <kfg>
      <arch>
        <bits>"64"</bits>
        <name>"56d19c8c9f15"</name>
      </arch>
      <dir>"/root/.ncbi"</dir>
      <name>"user-settings.mkfg"</name>
    </kfg>
    <libs>
      <cloud>
        <report_instance_identity>"true"</report_instance_identity>
      </cloud>
    </libs>
    <repository>
      <user>
        <ad>
          <public>
            <apps>
              <file>
                <volumes>
                  <flat></flat>
                  <flatAd>"."</flatAd>
                </volumes>
              </file>
              <refseq>
                <volumes>
                  <refseqAd>"."</refseqAd>
                </volumes>
              </refseq>
              <sra>
                <volumes>
                  <sraAd>"."</sraAd>
                </volumes>
              </sra>
              <sraPileup>
                <volumes>
                  <ad>"."</ad>
                </volumes>
              </sraPileup>
              <sraRealign>
                <volumes>
                  <ad>"."</ad>
                </volumes>
              </sraRealign>
              <wgs>
                <volumes>
                  <wgsAd>"."</wgsAd>
                </volumes>
              </wgs>
            </apps>
            <root>"."</root>
          </public>
        </ad>
      </user>
    </repository>
    <sra>
      <quality_type>"raw_scores"</quality_type>
    </sra>
    <vdb>
      <lib>
        <paths>
          <kfg>"/usr/local/bin"</kfg>
        </paths>
      </lib>
    </vdb>
  </Config>
  <RemoteAccess available="false"/>
  <CurrentProtectedRepository found="false"/>
 </Configuration>
 <Object path="https://sra-pub-run-odp.s3.amazonaws.com/sra/SRR5755657/SRR5755657" type="database" fs_type="unexpected">
  <Dependencies>
   <List count="22" missing="22">
    <Dependency index="0" seq_id="NC_000067.5" local="false" path=""/>
    <Dependency index="1" seq_id="NC_000068.6" local="false" path=""/>
    <Dependency index="2" seq_id="NC_000069.5" local="false" path=""/>
    <Dependency index="3" seq_id="NC_000070.5" local="false" path=""/>
    <Dependency index="4" seq_id="NC_000071.5" local="false" path=""/>
    <Dependency index="5" seq_id="NC_000072.5" local="false" path=""/>
    <Dependency index="6" seq_id="NC_000073.5" local="false" path=""/>
    <Dependency index="7" seq_id="NC_000074.5" local="false" path=""/>
    <Dependency index="8" seq_id="NC_000075.5" local="false" path=""/>
    <Dependency index="9" seq_id="NC_000076.5" local="false" path=""/>
    <Dependency index="10" seq_id="NC_000077.5" local="false" path=""/>
    <Dependency index="11" seq_id="NC_000078.5" local="false" path=""/>
    <Dependency index="12" seq_id="NC_000079.5" local="false" path=""/>
    <Dependency index="13" seq_id="NC_000080.5" local="false" path=""/>
    <Dependency index="14" seq_id="NC_000081.5" local="false" path=""/>
    <Dependency index="15" seq_id="NC_000082.5" local="false" path=""/>
    <Dependency index="16" seq_id="NC_000083.5" local="false" path=""/>
    <Dependency index="17" seq_id="NC_000084.5" local="false" path=""/>
    <Dependency index="18" seq_id="NC_000085.5" local="false" path=""/>
    <Dependency index="19" seq_id="NC_000086.6" local="false" path=""/>
    <Dependency index="20" seq_id="NC_000087.6" local="false" path=""/>
    <Dependency index="21" seq_id="NC_005089.1" local="false" path=""/>
   </List>
  </Dependencies>
 </Object>
 <SOFTWARE>
  <VDBLibrary vers="2.7.47"/>
  <Build static="true">
   <Module name=""/>
  </Build>
  <Tool date="Nov 18 2022" name="fastq-dump" vers="3.0.1">
   <Binary path="/usr/local/bin/fastq-dump" type="alias" md5="c461c39bfa514aff3c4f7c0416ced617">
    <Alias resolved="fastq-dump.3">
     <Alias resolved="fastq-dump.3.0.1">
      <Alias resolved="sratools.3.0.1"/>
     </Alias>
    </Alias>
   </Binary>
  </Tool>
 </SOFTWARE>
 <Env>
 </Env>
</Report>

Answer 1

Use the SRR accessions, not the sample-IDs you are using.

With entrez e-utils, I found the SRR of GSM2683458:

esearch -db sra -query GSM2683458 | efetch -format xml | xtract -pattern IDENTIFIERS -if PRIMARY_ID -contains SRR -element PRIMARY_ID is SRR5755657.

For GSM3603268, using esearch -db sra -query GSM3603268 | efetch -format xml | xtract -pattern IDENTIFIERS -if PRIMARY_ID -contains SRR -element PRIMARY_ID, the SRR is: SRR8571942

Then just do fasterq-dump SRR5755657; fasterq-dump SRR8571942.