A13 - Multiple Hypothesis Testing for Pre-screening of Factor Selection for Classification of High-Dimensional Data

Introduction: : Modern instrumentation, such as mass spectrometry, enables the simultaneous measurement of hundreds or thousands of compounds in individual samples. In many studies, these high-dimensional datasets are used to build classification models for applications ranging from assessing process performance to diagnosing patient health conditions. A common challenge is that the number of measured features far exceeds the number of available samples, increasing the risk of overfitting. Typically, having 10–20 samples per input variable is advisable, necessitating the pre-screening of compounds for statistically significant differences between groups. Simple methods, like a Student’s t-test, can yield about 5% false positives under the null hypothesis (p-value < 0.05) and established multiple hypothesis testing methods, such as the Bonferroni and Benjamini–Hochberg procedures, often prove overly conservative. To overcome these limitations, we propose a screening procedure that uses a Leave-n-Out (LnO) resampling strategy to better assess the robustness of p-values which enables more balanced feature pre-screening.

Materials and

Methods: : We developed a pre-screening approach that computes the false discovery rate of each variable’s p-value using a Leave-n-Out strategy. In this method, n samples are sequentially omitted from the dataset and the p-value is recalculated. This approach is first validated on synthetic datasets generated by varying the number of variables, the proportion of significant findings, and the extent of inter-variable correlation. Additionally, we applied the LnO method to two experimental datasets from metabolomics and metagenomics studies. Standard univariate hypothesis tests (e.g., t-tests or non-parametric alternatives) were performed on each dataset, and the results were compared to those obtained using Bonferroni and Benjamini–Hochberg corrections. Finally, candidate features identified by each method were further evaluated by constructing classification models using Fisher’s Discriminant Analysis, logistic regression, and support vector machines with cross-validation.

Results, Conclusions, and Discussions:: Our synthetic experiments revealed that, although the Benjamini–Hochberg procedure often achieved highest accuracy, the LnO approach provided comparable performance and outperformed the Bonferroni correction in many experiments—especially as the proportion of truly significant variables increased. In the metabolomics dataset, while traditional methods identified only two significant metabolites, the LnO method detected a substantially larger set of 17 candidate features. Classification models built on these LnO-selected features yielded cross-validated accuracies as high as 97–100%, compared to 84–87% using Bonferroni correction and Benjamini-Hochberg procedure-based selection. Similarly, in the metagenomic study, the LnO approach was the only method that provided significant variables to construct effective classifiers with up to 89.5% accuracy. Although the computational cost of the LnO method increases with larger n, its enhanced sensitivity in feature detection demonstrates its utility in high-dimensional settings. Overall, our findings indicate that pre-screening via LnO offers an alternative for identifying features that lead to improved classification performance in datasets with high dimensionality but low number of samples, such as often found in omics research.

Acknowledgements and/or References (Optional)::