Research

Detection coherence of tests

Some widely used tests are calibrated under a null narrower than the one their own test statistic implies. The gap between the two – the test’s blind spot – is a set of alternatives against which the test’s power does not grow with sample size, however large the sample: the test can neither reliably detect nor reliably rule out a real difference there. Detection coherence theory gives a set-theoretic framework for identifying which tests have this property and which do not.

The framework shows that the Wilcoxon–Mann–Whitney, Kruskal–Wallis, Friedman, and logrank tests – four of the most widely used nonparametric tests – are detection incoherent, while the Kolmogorov–Smirnov, Zaremba, and Maximum Mean Discrepancy tests are universally detection coherent.

Related work:

• Grendár, M. (2026). Detection coherence of tests. arXiv:2608.11177
• Grendár, M. (2025). Wilcoxon–Mann–Whitney Test of No Group Discrimination. arXiv:2511.20308 – the companion methods paper for the wmwAUC R package, recalibrating the WMW statistic under \(H_0\colon \mathrm{AUC} = 1/2\) instead of \(H_0\colon F = G\).
• A companion paper on detection coherence of two-sample survival tests – forthcoming.
• A companion preprint on missed discoveries by detection incoherent tests, aimed at a biomedical audience – forthcoming.

See also the project website for further material, including real-data illustrations and the Monte Carlo code underlying the framework’s examples.