0
$\begingroup$

I have all file names for each paired-end fastq pair as shown below. Could you please suggest how I can extract all fastq.gz files in a single folder?

sample name:

188833984]$ ls

Leaf_T1_FD_R10_L001-ds.3b884c360b1e4ae185408a613b90a3bc
Leaf_T1_FD_R2_L001-ds.7db8eb7e3426486db549426601b3a0bd
Leaf_T1_FD_R3_L001-ds.147177ecc03a46ccbbce3162d1185a0a

pair end file within sample name Leaf_T1_FD_R10_L001-ds.3b884c360b1e4ae185408a613b90a3bc

188833984]$ cd Leaf_T1_FD_R10_L001-ds.3b884c360b1e4ae185408a613b90a3bc

Leaf_T1_FD_R10_L001-ds.3b884c360b1e4ae185408a613b90a3bc]$ ls
Leaf-T1-FD-R10_S73_L001_R1_001.fastq.gz
Leaf-T1-FD-R10_S73_L001_R2_001.fastq.gz

I need all in fastq.gz format but the problem is that each sample, pair end R1 and R2 fastq reads in a single folder like this: and I need to extract all samples files in a single folder

$\endgroup$
4
  • 1
    $\begingroup$ Can you please edit your question and clarify. What do you mean by "extract all fastq files"? Do you just want to decompress (go from fastq.gz to fastq)? What is the final outcome you are expecting from the example you give? $\endgroup$
    – terdon
    Jul 1, 2019 at 12:15
  • $\begingroup$ I think OP wants to decompress fastq.gz files found in various subdirectories to fastq files in a single location. OP should probably decompress them where they are and create soft-links to them in their directory of choice. $\endgroup$
    – Ram RS
    Jul 1, 2019 at 14:18
  • $\begingroup$ @Dr.Yogesh, Do you want to copy/move all files to one folder or decompress them (as Ram RS is suggesting)? $\endgroup$
    – benn
    Jul 1, 2019 at 14:20
  • $\begingroup$ "this file name has R1 and R1 fastq.gz": what does this mean? Did you mean to write R1 and R*2*? You show a directory, Leaf_T1_FD_R10_L001-ds.3b884c360b1e4ae185408a613b90a3bc with two files, an R1 and an R2, so what is the problem? The files seem to already be in separate directories. Can you please edit and show us what the final result you want is? $\endgroup$
    – terdon
    Jul 1, 2019 at 14:29

1 Answer 1

3
$\begingroup$

Go to the common parent folder under which all FASTQ.GZ files reside. They could be in sub-folders under the folder, but all of them should have the folder you're in as a common parent folder. From such a folder, you should be able to run a find

find . -name "*.fastq.gz"

to get your FASTQ.GZ files. Now, there are surely better ways of doing this but here's one straightforward way.

Let's say you need to extract the FASTQs to ~/target/dir/, you'd do something like:

for f in $(find . -name "*.fastq.gz")
do
    tgt_file="~/target/dir/${f%.*}" #removed .gz from the name
    gzip -dc $f >$tgt_file #pigz -dc would be better if you have pigz
done

EDIT: OP has clarified that they wish to copy the files as such, not decompress them.

There are two ways to achieve this goal - you can either make physical copies of the files (thereby doubling the storage space you're currently using for them), or you can create symbolic links to the files (which are like shortcuts pointing to the actual files). I'll show you both ways here:

Let's assume the files are all under /fastq/files

find /fastq/files/ -name "*.fastq.gz" -exec cp {} ~/target/dir/ \; #This creates physical copies

cd ~/target/dir
find /fastq/files/ -name "*.fastq.gz" | xargs -I v_f ln -s v_f #These lines create symbolic links that point to the original files. Do not delete/move/rename the original files or these will become invalid.
$\endgroup$
3
  • $\begingroup$ Dear Ram, I want all subdirectory fastq.gz files into a single folder but files should be in fastq.gz format only, I don't want to decompress the files. $\endgroup$ Jul 1, 2019 at 15:19
  • $\begingroup$ I've updated my answer. Use symbolic links instead of copying the files. As long as you don't move the original files or rename them (or any of their parent directories), your links will work fine and take much less space than if you copied the actual files. $\endgroup$
    – Ram RS
    Jul 1, 2019 at 16:08
  • $\begingroup$ @Dr.Yogesh Please accept the answer if it worked for you. $\endgroup$
    – Ram RS
    Jul 2, 2019 at 2:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.