2
$\begingroup$

I have a dataset that contains thousands of mixed multiple and single fast5 files in a non-homogenous folder structure. The reads and fast5 files are coming from different labs and some labs gave single fast5 files and some multi fast5 files, without clear mappings. My final goal is to convert all the fast5 files to multi fast5 files.

There are commands in ont_fast5_api which is an interface to HDF5 files of the Oxford Nanopore. Using this API we can convert single fast5 files and multi fast5 files to each other. this is the link to this API github page: https://github.com/nanoporetech/ont_fast5_api

So I made a test mixed folder to check if I can use the API without missing reads as a result of conversion. The test folder structure is similar to my original folder structure:

  • mixed
    • Subfolder_A
      • single_fast5 x 4000
    • Subfolder_B
      • multiple_fast5 (contains 4000 single reads fast5)
    • single_fast5 x 2
    • multiple_fast5_1 (contains 4000 single reads fast5)
    • multiple_fast5_2 (contains 4000 single reads fast5)
    • multiple_fast5_3 (contains 100 single reads fast5)

Here is a screenshot of the structure with a few files from each folder:

alt text

and I created two other folders outside of the mixed folder as follows:

  • single

  • multi

Here my question is about single_to_multi_fast5 and multi_to_single_fast5 commands.

test_path=mixed
multi_path=multi
single_path=single
multi_to_single_fast5 -i $test_path/ -s $single_path/ --recursive

in the $single_path, I expect to see, all of the reads in multi files converted to single files (12100 reads in 12100 single fast5 files), but I see 8100 single files.

alt text So I have 4000 missing reads.

Now I convert single read fast5 file to multi:

single_to_multi_fast5 -i $test_path/ -s $multi_path/ --filename_base $output_name --batch_size 1000 --recursive

in the $multi_path folder, I expect to see that generated multi files contain all my single-read files (4000 + 2), but I see 3006 in the mapping summary.

alt text

So again I missed 996 single reads in my multi files. Since I made a small subsample of data, I am sure that the files are not overwritten.

Because my final goal is converting everything to multi I was going to do the following pipeline:

orig_path=original_folder
intermediate_path=intermediate
final_path=final_multi
# converts/collects all multi files in the original folder
multi_to_single_fast5 -i $orig_path/ -s $intermediate_path/ --recursive
single_to_multi_fast5 -i $intermediate_path/ -s $final_path/ --filename_base $output_name --batch_size 1000 --recursive

# converts/collects all single files in the original folder
single_to_multi_fast5 -i $orig_path/ -s $final_path/ --filename_base $output_name --batch_size 1000 --recursive

However because it seems I get missing reads during the process, I cannot use this pipeline.

At the same time, my original dataset is super huge, I cannot check if individual files are single or multi. It would take ages (I guess). Is there a solution for solving this problem besides checking every file? Maybe my pipeline is not quite suitable for what I want to do.

$\endgroup$
5
  • 1
    $\begingroup$ Can you please show the first few filenames in each folder? e.g. by running the following command: find mixed -type 'd' | while read dname; do echo "** ${dname} **"; tree -L 1 ${dname} | head; done $\endgroup$
    – gringer
    Dec 9, 2023 at 20:20
  • $\begingroup$ Thank you for your reply! I included the first few filenames in each folder @gringer $\endgroup$
    – Marjan
    Dec 10, 2023 at 2:56
  • 1
    $\begingroup$ The image seems like it's annotated incorrectly. A file with a channel and read number should be a single fast5 file, not a multi fast5 file. $\endgroup$
    – gringer
    Dec 10, 2023 at 5:25
  • $\begingroup$ Yes, the 2nd label (multi fast5 files with 4000 reads) and the 3rd label (single fast5 files) should be swapped. @gringer $\endgroup$
    – Marjan
    Dec 13, 2023 at 21:06
  • 1
    $\begingroup$ Okay, that's good to know; thanks for the clarification. $\endgroup$
    – gringer
    Dec 14, 2023 at 2:44

1 Answer 1

1
$\begingroup$

It looks like you're creating too much additional work for yourself. What I'm trying to lead to with my comments is that the file names should help in working out if a file represents single reads or multiple reads, in which case the single reads can be shifted into a separate folder.

Based on the file names that I would expect, a search for files containing '_read_[0-9]' should be sufficient to fish out the single fast5 files:

find . -name '*_read_[0-9]*' > single_reads.txt
mkdir -p single_reads
cat single_reads.txt | while read fname; do mv -i ${fname} single_reads; done

[That single_reads.txt file could be split into multiple files if directory entries get too large.]

After which you should have a single folder that contains thousands of single-read fast5 files, and other folders that contain multi-fast5 files, removing the problem of needing to deal with mixed file types.

$\endgroup$
1
  • 1
    $\begingroup$ I am gonna try this, thank you! @gringer $\endgroup$
    – Marjan
    Dec 13, 2023 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.