Skip to main content

ARE DISCOVERIES SPURIOUS? DISTRIBUTIONS OF MAXIMUM SPURIOUS CORRELATIONS AND THEIR APPLICATIONS.

Author(s): Fan, Jianqing; Shao, Qi-Man; Zhou, Wen-Xin

Download
To refer to this page use: http://arks.princeton.edu/ark:/88435/pr1kk38
Full metadata record
DC FieldValueLanguage
dc.contributor.authorFan, Jianqing-
dc.contributor.authorShao, Qi-Man-
dc.contributor.authorZhou, Wen-Xin-
dc.date.accessioned2021-10-11T14:17:39Z-
dc.date.available2021-10-11T14:17:39Z-
dc.date.issued2018-06en_US
dc.identifier.citationFan, Jianqing, Shao, Qi-Man, Zhou, Wen-Xin. (2018). ARE DISCOVERIES SPURIOUS? DISTRIBUTIONS OF MAXIMUM SPURIOUS CORRELATIONS AND THEIR APPLICATIONS.. Annals of statistics, 46 (3), 989 - 1017. doi:10.1214/17-aos1575en_US
dc.identifier.issn0090-5364-
dc.identifier.urihttp://arks.princeton.edu/ark:/88435/pr1kk38-
dc.description.abstractOver the last two decades, many exciting variable selection methods have been developed for finding a small group of covariates that are associated with the response from a large pool. Can the discoveries by such data mining approaches be spurious due to high dimensionality and limited sample size? Can our fundamental assumptions on exogeneity of covariates needed for such variable selection be validated with the data? To answer these questions, we need to derive the distributions of the maximum spurious correlations given certain number of predictors, namely, the distribution of the correlation of a response variable Y with the best s linear combinations of p covariates X, even when X and Y are independent. When the covariance matrix of X possesses the restricted eigenvalue property, we derive such distributions for both finite s and diverging s, using Gaussian approximation and empirical process techniques. However, such a distribution depends on the unknown covariance matrix of X. Hence, we use the multiplier bootstrap procedure to approximate the unknown distributions and establish the consistency of such a simple bootstrap approach. The results are further extended to the situation where residuals are from regularized fits. Our approach is then applied to construct the upper confidence limit for the maximum spurious correlation and testing exogeneity of covariates. The former provides a baseline for guarding against false discoveries due to data mining and the latter tests whether our fundamental assumptions for high-dimensional model selection are statistically valid. Our techniques and results are illustrated by both numerical examples and real data analysis.en_US
dc.format.extent989 - 1017en_US
dc.languageengen_US
dc.language.isoen_USen_US
dc.relation.ispartofAnnals of statisticsen_US
dc.rightsAuthor's manuscripten_US
dc.titleARE DISCOVERIES SPURIOUS? DISTRIBUTIONS OF MAXIMUM SPURIOUS CORRELATIONS AND THEIR APPLICATIONS.en_US
dc.typeJournal Articleen_US
dc.identifier.doidoi:10.1214/17-aos1575-
dc.identifier.eissn2168-8966-
pu.type.symplectichttp://www.symplectic.co.uk/publications/atom-terms/1.0/journal-articleen_US

Files in This Item:
File Description SizeFormat 
Are discoveries spurious Distributions of maximum spurious correlations and their applications.pdf1.1 MBAdobe PDFView/Download


Items in OAR@Princeton are protected by copyright, with all rights reserved, unless otherwise indicated.