0
$\begingroup$

Using Crowell 52. When I do a glob to get matching files, Cromwell places a copy of each matching file in the glob-* directory.

As these are multi-gigabyte Bam files, I'd really prefer for Cromwell to store soft links to the files, rather than full copies.

The code:

   output {
      Array[File] Outputs = glob("*.fastq.gz")
   }

Is it possible to make this happen?

$\endgroup$
2
  • 1
    $\begingroup$ It would help your case to show the code you're using. $\endgroup$
    – Ram RS
    Mar 9, 2021 at 15:51
  • $\begingroup$ @RamRS added above $\endgroup$ Mar 10, 2021 at 15:12

1 Answer 1

1
$\begingroup$

As stated in https://github.com/broadinstitute/cromwell/issues/4395

The problem may be because you can't hardlink folders on your system for various reasons (i.e. filesystem doesn't allow hardlinking folders, only files, or you are trying to link between different filesystems).

You can try overriding the default way glob works.

i.e. add this to your cromwell.config

glob-link-command = "ls -L GLOB_PATTERN 2> /dev/null | xargs -I ? ln -s ? GLOB_DIRECTORY"

Like this

backend
{
  default = "Slurm"
  providers
  {
    Htcondor {
      actor-factory = "cromwell.backend.impl.sfs.config.ConfigBackendLifecycleActorFactory"

      config
      {
        glob-link-command = "find GLOB_PATTERN 2> /dev/null | xargs -I ? ln -s ? GLOB_DIRECTORY"
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.