# Multithread fastq processing with kseq.h in C++11?

## Background

I am using the wonderful kseq.h as my fastq/fasta parser for a C++ project. My actual implementation uses something like this, where I am counting kmers of each sequence and performing a bunch of other processing on every read pair.

Thanks to @user1782818's code my program is very fast, but I want it to be faster. The only way I know how to do this is with multithreading. However, I have never worked with multithreading in C++.

For each read pair I am accessing a vector or map of structs. Within each struct I am incrementing counters or pushing values onto vectors. Essentially I am doing something like this:

//compiled with g++ 7.3 using command g++ this_prog.cpp -o this_prog -lz
// execute with ./this_prog my_file.fastq.gz
#include <iostream>//cout
#include <map>
#include <zlib.h>
#include <stdio.h>
#include "kseq.h"

int main(int argc, char **argv)
{
//initialize prefix
std::map<std::string, int> prefix_counter;
std::string prefix;
prefix.resize(2);

gzFile fp;
kseq_t *seq;
int n = 0, slen = 0, qlen = 0;
fp = gzopen(argv[1], "r");
seq = kseq_init(fp);
++n;
//COUNT PREFIXES
std::copy(seq->seq.s, seq->seq.s + 2, begin(prefix));
if (prefix_counter.count(prefix) == 0){
prefix_counter[prefix] = 1;
} else prefix_counter[prefix]++;
}
std::cout << "Total n. of sequences: " << n << '\n';
kseq_destroy(seq);
gzclose(fp);

//print counts
for( const auto& sm_pair : prefix_counter ){
std::cout << sm_pair.first << ": " << sm_pair.second << '\n';
}
return 0;
}


Example output from the above program is something like this for a 100,000-read test fastq file.

Total n. of sequences: 100000
AA: 8561
AC: 5178
AG: 7634
AT: 8523
CA: 7526


## Question

Q1: Is it possible to perform a task like the one shown above using multithreading and kseq.h to read in a fastq file and process the reads in chunks? I only want to use standard libraries. C++11 is OK - I know that multithreading support was added in C++11.

Q2: Can you please provide a C++ example of how to convert the above or similar program to use multithreading using kseq.h?

• Note: I'm not posting this on SO to avoid "read the manual"-type responses and I believe that others in the bioinformatics SE community will benefit from an answer to this question. Thanks! – conchoecia Dec 20 '18 at 19:15

I'll note that my library relies on a header-only MPMC implementation and so technically isn't "only using the standard library", but you're already relying on kseq.h, so omitting third-party libraries must not be a hard constraint.