2
$\begingroup$

I have a folder where the hdf5 files inputs are (matrices of scRNAseq raw UMI counts). There is a python script that loads them, splits them into multiple matrices, normalizes them, and saves as csv files. When the name pattern of the output files is known, things are easy, I know how to do that, but when one input (hdf5 file) can produce various number of outputs (csv files) I am not sure how to approach the issue. Currently my Snakemake file relevant part looks the following way:

hdf5_files,=glob_wildcards('/main/{hdf5_file}.hdf5') 

rule all:
  input:
--->  expand('data_files/normalized/{hdf5_file}.csv', hdf5_file=hdf5_files)
    ...

rule normalization:
  input:
     script = 'python/normalize.py',
     path = '/main/{hdf5_file}.hdf5'
  output:
--->  path = 'data_files/normalized/{hdf5_file}.csv'
  shell:
     "python {input.script} -data {input.path} -path {output.path}"

Above I highlighted the 2 lines that I am not sure now how I should write. Any suggestions would be greatly appreciated.

$\endgroup$
5
  • $\begingroup$ Have you considered using flag files? Or do you need the names for downstream processing too and can't use glob or specify a directory at that point? $\endgroup$
    – Devon Ryan
    Sep 5, 2018 at 19:55
  • $\begingroup$ The rule normalization processes, say, 2 hdf5 files and outputs 10 csv ones into data_files/normalized folder. Then, I want to use glob_wildcards to read in these 10 csv files and move on. Further rules are, of course, fully dependent on the rule normalization because it generates the input csv files. I am not sure how can I use flag files, but will try. Thank you! $\endgroup$ Sep 5, 2018 at 21:39
  • 2
    $\begingroup$ Ah, if you need to process each of the resulting csv files independently downstream then there are no snakemake tricks that I know of that will help (I presume glob_wildcards() is run during rule parsing, so the CSV files won't be there yet). I suspect you'll want to run normalization from a wrapper script beforehand in that case, since then globbing will be certain to work as desired. $\endgroup$
    – Devon Ryan
    Sep 5, 2018 at 22:19
  • 1
    $\begingroup$ What do you mean from a wrapper script? Outside the pipeline, you mean? $\endgroup$ Sep 6, 2018 at 2:27
  • 2
    $\begingroup$ Yes, exactly. A (probably python) script that runs two Snakefiles when you call it, with appropriate options. Then you have the flexibility you need while maintaining the convenience of snakemake. $\endgroup$
    – Devon Ryan
    Sep 6, 2018 at 7:04

1 Answer 1

4
$\begingroup$

Could you give an example of names of input files and output files of normalize.py ?

It seems to me that your case could be similar to the one described about the dynamic output:

https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#dynamic-files

Here you would probably use it for the output of normalization and for the input of all (with a wildcard different from hdf5_file).

Disclaimer: I haven't had the opportunity to use this feature so far.

$\endgroup$
1
  • $\begingroup$ Sample name is {hdf5_file}_{num}.hdf5, where num is an integer. $\endgroup$ Sep 11, 2018 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.