How can I develop a snakemake rule for all files in multiple directories?

Question

I am a beginner in snakemake and I'm trying to develop a working flow.

Imagine I have multiple folders and in each folder I have a tumour and normal sample files:

SRC120/ 
      SRC120-T1.fastq
      SRC120-N1.fastq

SRC121/ 
      SRC121-T1.fastq
      SRC121-N1.fastq

and so on (in total I have more than 100 SRC folders).

I am trying to make a snakemake rule to make a symlink of these fastq files.

What I have written so far is:

import os

SAMPLES=os.listdir("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/")                                               

def getFastqFilesForTissu(wildcards):
    fastqs = list()
    # Beware no other files than fastqs should be there
    for s in os.listdir("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/"+wildcards.sample):
        fastqs.append(os.path.join("/analysis/Anna/snakemake-demo/data/samples_fastq-test1/",wildcards.sample,s))
    return fastqs


rule all:                                                                        
    input:                                                                       
        expand("results/{sample}/{sample}.fastq", sample=SAMPLES)                   


rule foo:
    input:
        fastq_files = getFastqFilesForTissu 
    output:
        "results/{sample}/{sample}.fastq"
    shell:
        'ln -s {input} {output}'

This works if I have one file in each folder and when the file name matches with the folder's name (i.e., when there is no T1 or N1 suffix).

Is there any way I could modify my rule to get it working for all samples in SRC directories? Or a better way to handle this issue?

Answer 1

I suggest using package glob, which supports unix-style pattern search. For instance finding all .py files in my repo:

import glob 
all_python_files = glob.glob(r'c:\personal\coderepo\**\*.py', recursive=True)

Which outputs full paths:

'c:\personal\coderepo\SimpleHTR\data\checkDirs.py',
'c:\personal\coderepo\....

I typically like to input the glob.glob to Pandas dataframes:

all_files = glob.glob(f'{base_folder}/*')
all_files = pd.DataFrame(all_files, columns=['full_path'])
all_files['my_filename'] = all_files['full_path'].apply(lambda x: os.path.basename(x))

#do whatever filtering is required:
all_files = all_files[all_files['my_file_name'].str.contains('regex_here')]

Then perform what ever I need to do in loop:

for idx, _ in all_files.iterrows():
    do_something(_['full_path'])