Why is the p-value significance threshold for HLA association tests $5*10^{-8}$? [closed]

Question

Typically, a p-value of $p<=5*10^{-8}$ is used for genome-wide association testing, as there are roughly $1/p$ independent LD blocks in the human genome, so it correct for multiple testing across the genome.

HLA assocation tests, such as in this study, only look at a fraction of the genome, something like chr6:28–34 Mb, which is approximately 0.2% of the genome used to conduct a true 'genome-wide' association study. However, they still use the same genome-wide p-value threshold of $p<=5*10^{-8}$.

Is this not too conservative, since there is a much reduced multiple testing burden? Should the threshold not be something like $p/0.002$?

Answer 1

Its a very good question and it is also a good study. What you are saying must be correct.

I thought it was possible they used the correct critical probability in the original submission and a reviewer requested it was changed. However I checked their medRxiv upload here and the authors clearly use this threshold prior their formal submission.

We performed stepwise conditional analysis 479 to identify additional independent signals by adjusting for the most significant amino acid 480 position in each step until none met the significance threshold ( P = 5 10^-8)

Thus I don't know.

If this paper is central to your area of interest I would use the corrected critical probability threshold and re-assess and extend their conclusions because 5 decimal places is a huge a difference.

Answer 2

As a direct answer to your question, I would argue that the authors, in choosing this threshold, have demonstrated a lack of understanding of their process of analysis. But that answer is not a full answer, and it's a problematic answer because [almost] everyone who does population genetics does similar things. This p-value is not appropriate, not merely because it is at a genome-wide significance level, but because no threshold is appropriate.

To explain my perspective more, thresholding to identify "statistically-significant" results, in general, is not a good approach. Regardless of what threshold is used, p-values only indicate the confidence in a statistic's value (as it relates to fitting a particular statistical model), not importance or the size of the value.

The authors of this paper, as with many other authors who have written similar papers, make an obvious statistical error in the legend of figure 3, where they state: "One-field classical allele HLA-B*57 (P = 9.84 × 10−138) (a) and amino acid position 97 in HLA-B (Pomnibus = 1.86 × 10−184) (b) showed the strongest association signal." This is an error, because they have only reported here the p-values, and p-values have no relevance to the strength or magnitude of a statistic.

Given this, it doesn't make sense to rank by p-value, and it especially doesn't make sense to display p-values on a graph and highlight p-value peaks as indicating association peaks. Those regions are confidence peaks.

In an attempt to provide some constructive light around this, I will answer the obvious follow-up questions are "How should p-values be used?" and "If p-values shouldn't be displayed or ranked, then what should?"

My preference is to use p-values as a loose filter, to exclude obviously incorrect results. In the case of a GWAS, I might, for example, choose to exclude any statistic that has a p-value above 0.1, leaving the remainder as potentially correct. It could possibly be argued that this is still thresholding. It is, in a sense, but importantly I am not saying that all values that pass the threshold are good, just that the ones that are excluded are all bad; I allow for some rubbish to end up in the good pile. I liken it to using a computer to find the things that look like diamonds, so that I don't waste my time inspecting dirt.

After that filtering is done, the p-values should be ignored, and the descriptive statistic used to rank, determine significance, association, or whatever the goal of the analysis is. For GWAS, a commonly-used value is beta, which can be positive or negative depending on the direction of association, and relates to the variant difference between groups.

If a list needs to be cut down, then this descriptive statistic should be used for thresholding, depending on the analysis. For example, it might be appropriate to display the 100 most extreme beta values, or threshold to exclude any beta values that have a magnitude less than 0.01, or display a manhattan-like plot with beta on the $y$ axis, and chromosome position on the $x$ axis.

There shouldn't be set rules, because every situation is different - there's a human element to this. What I'm saying is that it should be the job of the researcher, not the computer, to identify which data are significant. Make the data set small enough that it can be processed by a person, then let them do the rest. I think Hannah Fry explains this quite well in this talk:

Now this is the stuff the algorithms are amazing at: looking for tiny, tiny clues in seemingly completely disconnected data sets.... Just incredibly, incredibly sensitive these algorithms, right? Just absolutely amazing what they can do. And yet remember that sensitivity does not make a perfect algorithm. That's only one half of the equation.... [Humans] are terrible at sensitivity, right? We are really, really bad. We miss things all the time. But, specificity, being specific, that's like our superpower.... And rather than choosing between human and machine, which is kind of the rhetoric that we get so much, why don't we exploit each other's strengths and just create much more of a partnership?.... Because the algorithm never gets tired, so let it trawl through all of that data and just highlight a few key areas of concern. And then the human never misdiagnoses. So they can come in, and just sweep up and have the final say.