THE SPECTRUM OF RANDOM INNER-PRODUCT KERNEL MATRICES

Abstract

We consider nxn matrices whose (i, j)th entry is f(X-i(T) X-j), where X-1,..., X-n are i.i.d. standard Gaussian in R-p, and f is a real-valued function. The weak limit of the eigenvalue distribution of these random matrices is studied at the limit when p, n -> infinity and p/ n = gamma which is a constant. We show that, under certain conditions on the kernel function f, the limiting spectral density exists and is dictated by a cubic equation involving its Stieltjes transform. The parameters of this cubic equation are decided by a Hermite-like expansion of the rescaled kernel function. While the case that f is differentiable at the origin has been previously resolved by El Karoui [The spectrum of kernel random matrices, Ann. Statist. 38 (2010) 1-50], our result is applicable to non-smooth f, such as the Sign function and the hard thresholding operator of sample covariance matrices. For this larger class of kernel functions, we obtain a new family of limiting densities, which includes the Marcenko-Pastur (M.P.) distribution and Wigner’s semi-circle distribution as special cases.