12

Proximity to NCBI may not necessarily give you the fastest transfer speed. AWS may be deliberately throttling the Internet connection to limit the likelihood that people will use it for undesirable things. There's a chance that a home network might be faster, but you're likely to get the fastest connection to NCBI by using an academic system that is linked ...


9

samtools quickcheck is all you need. From the manual: Quickly check that input files appear to be intact. Checks that beginning of the file contains a valid header (all formats) containing at least one target sequence and then seeks to the end of the file and checks that an end-of-file (EOF) is present and intact (BAM only). Data in the middle of the file ...


8

-O is to specify the name of the output file. You want the -P option. So, from your example wget -b "ftp://ftp.ncbi.nlm.nih.gov/blast/db/nt.??.tar.gz" -P output/


7

I don't know whether there is an API, but ENCODE's website does provide an interactive data matrix where you can filter data based on assay and sample type, place data sets in a "shopping cart", and then proceed to "checkout" to download the files of interest.


6

In general, the best way to download SRA data is: don't download from SRA. However, as ENA has not be sync'd yet, I would recommend to download from SRA ftp and then convert to fastq locally. You can find files in the SRA format here. Downloading and then converting locally is much faster than direct retrieval from NCBI for some mysterious reasons.


5

Finally, I found an alternative to the SRA translation: a link that works! For those of you interested in knowing how to download FastA files from NCBI using an accession number, try the following link: https://www.ncbi.nlm.nih.gov/search/api/sequence/${accession}/?report=fasta Using wget to download the accession used as example: wget -O NC_001416.1....


5

You can use the mirroring function of wget: wget -m ftp://ftp.ensembl.org/pub/release-91/embl/bos_taurus/


5

awk '{print "wget " $9 " -O " $1 "_" $2 ".fastq.gz"}' tmp will give you wget https://www.encodeproject.org/files/ENCFF001JPX/@@download/ENCFF001JPX.fastq.gz -O a_k.fastq.gz wget https://www.encodeproject.org/files/ENCFF001JPZ/@@download/ENCFF001JPZ.fastq.gz -O b_k.fastq.gz wget https://www.encodeproject.org/files/ENCFF001JQB/@@download/ENCFF001JQB.fastq....


5

You can just add /?format=json to any page to get the JSON output. ENCODE REST API documentation: https://www.encodeproject.org/help/rest-api/ Example scripts: https://github.com/ENCODE-DCC/submission_sample_scripts


4

A variation of Bioathlete’s answer that doesn't suffer from escaping issues: awk 'BEGIN{ FS="\t" } { printf("%s\t%s_%s.fastq.gz\n", $8, $1, $2) }' input | while IFS=$'\t' read -r url filename; do wget -O "$filename" -- "$url" done Or entirely with shell parsing: while IFS=$'\t' read -r c1 c2 c3 c4 c5 c6 c7 url; do wget -O "${c1}_${c2}.fastq.gz" -- ...


4

A quick look at your link tells me the SRR numbers run from SRR837819 to SRR837856. You can use fastq-dump from the sratoolkit, and make a for loop around it in bash. Something like this should work: for (( i = 19; i <= 56; i++ )) do fastq-dump --accession SRR8378$i done After reading Devon Ryan's answer, I realize that you asked for SRA files ...


4

The Persistent uniform resource locator or PURL is one such solution, these are designed to be a bit more robust than permalinks in so much as they are supposed to survive the change of domain name. The bio ontology community already use them http://purl.bioontology.org/docs/index.html


4

A sample code is given in the salmon documentation as follows. Source #!/bin/bash mkdir data cd data for i in `seq 25 40`; do mkdir DRR0161${i}; cd DRR0161${i}; wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/DRR016/DRR0161${i}/DRR0161${i}_1.fastq.gz; wget ftp://ftp.sra.ebi.ac.uk/vol1/fastq/DRR016/DRR0161${i}/DRR0161${i}_2.fastq.gz; cd ..; done cd .....


3

You can download raw data from SRA (NCBI). On this website under RNA-Seq alignments, you'll find the samples. Click on them, for example SAMN05231885, and then click on PRJNA325427 to see the link to the SRA data (click on 1). Here you can download the raw fastq files of the experiment.


3

By far the fastest method in my experience has been to use the SRAdb library in R. For most entries, you can download fastq files directly. Some older experiments don't have them, but I've still found it much faster to download SRA files via getSRAfile() and then to convert them using fastqdump than to use fastqdump directly.


3

Turns out you can just grab the GFF3 from the NCBI's FTP site! ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/003/195/GCA_000003195.3_Sorghum_bicolor_NCBIv3/ See: https://www.ncbi.nlm.nih.gov/genome/doc/ftpfaq/#files Or "Access the data" on the right here: https://www.ncbi.nlm.nih.gov/assembly/GCF_000003195.3


3

Assuming you ultimately just want the fastq files and you know the SRR (run) numbers, I would download them from here: ftp://ftp.sra.ebi.ac.uk/vol1/fastq/ As for downloading multiple files, I've just used multiple wget commands. I don't know of a way to download all of the files together in like a zipped folder or anything :/


3

I suggest you follow the advice in Eric A Brenner's answer and just download the fastq files. However, if you really really want to use the SRA files for some reason, note that you can use parallel-fastq-dump to make things faster. Do follow its advice regarding using prefetch. You'd need to combine that with the answer from b.nota (i.e., put the commands ...


3

I presume you are looking for the recently sequenced WGS data rather than the genotype data? If so, the different files can be found here: ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/ Each chromsome has a seperate vcf file with a corresponding .tbi index file. The gvcfs can be found here: ftp://ngs.sanger.ac.uk/production/hgdp/hgdp_wgs.20190516/...


2

I would recommend to do this in R, using biomaRt. For example if you want all the SNPs from IDH1 (ENSG00000138413) gene: library(biomaRt) ensembl<- useMart("ensembl") ensembl <- useDataset("hsapiens_gene_ensembl",mart=ensembl) IDH1 <- getBM(attributes=c('ensembl_gene_id','variation_name'), filters = 'ensembl_gene_id', values ="ENSG00000138413", ...


2

The HapMap website was suspended due to a security issue. Archived HapMap data is available via FTP, but it's a better idea to download up-to-date International Genome data from the IGSR website (which includes the HapMap individuals as a subset). The format you use will be dependent on the data analysis program you're using. In most cases involving SNP ...


2

The filenames of those bams contain the cell line names. They are constructed: SAMPLEID.CELL_LINE_NAME.REPLICATE.bam Thus the first 5 cell lines in your screen capture are SK-N-F1, TE-9, CJM, HCI-H1915 and CAL-33. All the information that is need is also present in the experimental metadata xml file attached to each sample record. This is accessible by ...


2

Here is a Biopython solution via Entrez utilies, simply copy the code into a python script and execute it: from Bio import Entrez, SeqIO Entrez.email = 'me@email.com' def get_fasta_from_ids(ids_list): handle = Entrez.efetch(db='nucleotide', id=','.join(ids_list), rettype='fasta') for record in SeqIO.parse(handle, 'fasta'): yield record ...


2

So let's say I want to get the bed file of this experiment using curl encodeproject.org/experiments/ENCSR000CKC You can also search for files that belong to a given experiment, e.g. https://www.encodeproject.org/search/?type=File&dataset=/experiments/ENCSR000CKC/ Then further select file properties (status, format, etc.) using the facets on the left. ...


2

So far, grabseqs is the easiest option, a wrapper for fastq_dump and fasterq-dump, you can install it with conda, I recommend you to use an environment: conda create -n grabseqs -c louiejtaylor -c bioconda -c conda-forge grabseqs conda activate grabseqs grabseqs sra SRRNNNNNNNN conda deactivate And download directly projects, runs, and bioprojects from ...


1

I would recommend using UCSC Xena instead of GDC for CCLE data. They don't have everything. For example, BAM files are not available. However, the data that is available is organized in standard text files with consistent identifiers.


1

You can either do this directly as you show in your answer or, for a more sophisticated and flexible approach, use NCBI's edirect tool: esearch -db nucleotide -query 'NC_001416.1' | efetch -format fasta > NC_001416.1.fa


1

Here are two python functions I use for getting stuff from ENA based on an accession: import os import itertools from urllib.request import urlopen def fetch_ENA_files(accession): """Get the names of the files matching the ENA accession""" # Get the paths to the files url = "http://www.ebi.ac.uk/ena/data/warehouse/filereport?accession=%s&...


1

Apparently there is no 100% reliable solution, but in the most cases, you should be able to use the enaBrowserTools python scripts (enaDataGet) to quite reliably download data. The run ERR2135453 is a special case because the sample submission (ERS1939590) for it has been cancelled. The solution for these reads is to download the data manually from the ftp ...


1

These data are in Experiment Hub in Bioconductor library(curatedTCGAData) brcaRNA <- curatedTCGAData("BRCA", "RNASeq2GeneNorm", dry.run = FALSE)


Only top voted, non community-wiki answers of a minimum length are eligible