Snakemake and github actions

Question

I am working on a snake-make workflow that follows the guidelines of formatting and linting so that it can be published on the official Snake-make repository. Whenever I push my code on GitHub, the GitHub actions are run and the pipeline is tested as indicated here.

The problem is related to the execution of some time-consuming rules, such as the genome indexing generation for bowtie. These rules are run every time and take plenty of time. I was wondering whether there was a way to bypass the execution of these rules and test the pipeline anyway. Do you have any suggestions about this?

A problem is that I have a rule in my snakemake workflow that downloads the human genome and that is needed to index and then for the alignment. I need a workaround to avoid this behaviour, since has you said it is not feasible to test on the whole genome.

Thanks

Answer 1

The answer for any testing purpose is generally to put together a small and minimal dataset so the test completes in seconds or minutes while still being representative. It makes (to me) limited sense to have a Actions/CI running using for example the entire human genome as input. The point of testing is to check that steps are executed properly, arguments are passed correctly, errors are detected. For this use either a small genome (such as from a bacterium or yeast), or make an artificial one consisting of very few basepairs.

What I often do (for example for a peak calling workflow) is to take a full dataset, then subset to a region where I see a) many peaks and b) no peaks, then extract this genomic range (a few thousand bp) to build an artificial genome, plus extract from the bam/fastq the reads contributing to this region. That way I have basically a read-world dataset but very limited in size. For CI testing that does very well and CI finished in seconds, allowing to test even different configurations of this workflow on a standard / unpaid Actions runner.

I personally prefer to not automate downloads but rather to have the user provide an actual file, or provide the file pre-downloaded in the workflow bundle. Connections might be bad or broken that day or the server/HPC is not connected to internet, and the entire workflows fails. Maybe write the relevant section in a way that it either accepts a URL for download OR a local file, and then write a testing profile that then reads the file to ensure it always is accessable to the Actions runner.