How does Li and Durbin's BWA paper compare alignment programs on real data?

Question

Li and Durbin's "Fast and accurate short read alignment with burrows-whleeler transform" found here, says:

We evaluate the performance of BWA on ... real paired-end data by checking the fraction of reads mapped in consistent pairs and by counting misaligned reads mapped against a hybrid genome.

The data for the real genome aligned to is in table 2.

Which pairs are referred to here? Are these the different software tools aligning the same corresponding reads meaning that they aligned the same sequences to similar locations? Are these paired ends? Or is this something different?

Answer 1

For the simulated reads, they know exactly where the reads came from, because they are able to create that link at the time paired-end reads are generated from the genome. Those simulated reads would be created through a process similar to the following:

1. Pick a single position in the genome at random
2. Pick another region in the same chromosome that is <fragment length>+error bases away from the first location
3. Extend these positions by <read length> (one read in the forward direction, and the other read in the reverse complement direction) to generate a read pair
4. Change at random some bases (or none) within the reads to simulate sequencing error
5. store the read pair in a FASTA file, together with the original position
6. repeat steps 1-5 a few hundred million times

The effectiveness of the algorithm would be based on how good the algorithm was at picking the correct genomic location.

For the real paired-end data, it's a classical mapping problem: there's no ground truth, so all that can be done is starting at step 5, and working back to all the possibilities for step 1. If the two sequences from a read pair are derived from the same fragment, they should map to approximately the same genomic location and [usually] in a reverse-complement orientation to each other. Such an alignment would be called "consistent", or "properly mapped".

Answer 2

What they are pairing is the RNAseq data with the reference genome: as the legend of the table 2 says:

The 12.2 million read pairs were mapped to the human genome

These softwares align the same read to the same genome. They aren't comparing if it is mapped to the same location, only how many mismatches has each program.