Inside the Subspace Where Spurious Correlations Are Born | Towards Data Science
between cholesterol and blood pressure is measured in 10 patients. The observed correlation is 0. 62. Is this enough evidence that a relationship exists?Studies that measure 20,000 gene expression levels in 10. . .
between cholesterol and blood pressure is measured in 10 patients. The observed correlation is 0. 62. Is this enough evidence that a relationship exists?Studies that measure 20,000 gene expression levels in 10 mice routinely apply multiple-testing corrections and stringent correlation thresholds. Researchers know that the more correlations they examine, the more likely they are to observe large values purely by chance, even when the genes are completely unrelated. With small studies, researchers often assume they are safer from spurious correlations. Are they?The distribution of a sample correlation is the same regardless of the total number of variables measured. It depends primarily on the number of subjects. High-dimensional datasets do not create spurious correlations—they simply make them easier to encounter. This article answers the question: what correlation values should we expect when the variables are independent and the true population correlation is zero? It also explains why such correlations occur. Using the geometry of Pearson’s correlation coefficient, the article visualizes the effects of centering and normalization. It then discusses the role of rotational invariance. Together, these ideas build intuition for the exact distribution of Pearson’s correlation coefficient and its asymptotic normal approximation. Finally, the article offers practical guidance on how to use this intuition when reading and conducting research. The code to reproduce all figures and experiments is available in the accompanyingnotebook. The Gaussian ExperimentImagine measuringdvariables onnindividuals. To study correlations that arise purely by chance, consider a simulated dataset whose columns are independent Gaussian random vectors of dimensionn. The resulting feature smatrix, where rows correspond to subjects and columns to variables, iswith columns satisfyingWhat is the sample correlation between two vectors?Even though the population correlation between any pair of variables is exactly zero, the sample correlations we observe are almost always nonzero. This raises a question: when the columns are truly independent, what values should we expect for the sample correlation?The sample (Pearson) correlation between columnskandliswhereis the sample mean of thek-th column. The vector of all column means isBecause the columns are random vectors, the sample correlation r(X⋅k, X⋅l)r\,(X_{\cdot k},\ X_{\cdot l}) is itself a random variable taking values in [−1,1]\left[-1, 1\right]. The intuition behind its derivation becomes clearer by understanding how Pearson’s correlation coefficient transforms the data. Step 1: centering moves vectors onto a hyperplaneThe numerator is effectively centering the vectors. Centering subtracts the sample mean from each columnwhere 𝟏n\mathbf{1}_n is the column vector of onesAfter centering, the coordinates of every column sum to 0This means every centered column now lies in the (n-1)-dimensional subspaceperpendicular to the vector of ones . The vectors remainn-dimensional, but after centering they are restricted to an(n-1)-dimensional hyperplane. They no longer vary in the direction of the all-ones vector. Sample sizen = 3, centering constrains all vectors to a planeForn = 3, the hyperplane is an ordinary plane in three-dimensional space. Rotating the plane to align with thexy-plane makes it visible: there is no variability in thez-direction, and the vectors are restricted to a two-dimensional subspace. Sample sizen = 4: centering constrains all vectors to a three-dimensional hyperplaneAfter choosing a coordinate system orthogonal to the all-ones vector, the centered vectors can be visualized in three dimensions. They form a cloud centered at the origin. Step 2: normalizing moves vectors onto the unit sphereNotice that Pearson’s correlation is exactly the cosine of the angle between two centered vectorsThe distribution of the sample correlation between two independent random vectors is therefore the same as the distribution of the cosine of the angle between their centered versions. When the centered columns are normalizedthe correlation matrix simplifies toCentering moved the vectors onto an(n−1)-dimensional hyperplane, removing information about location. Normalizing removes differences in lengthremoving information about scale. The vectors are now constrained to the unit sphere embedded in the(n-1)-dimensional hyperplane, an(n−2)-dimensional manifoldn = 3: normalizing moves vectors onto the unit circleForn = 3, centering moved the vectors onto a plane. Normalizing fixes their length at one, restricting them to the one-dimensional sphere within that plane—the unit circle. n = 4: normalizing moves vectors onto the unit sphereForn = 4, centering moved the vectors into a cloud centered at the origin within a three-dimensional subspace. Normalizing fixes their length at one, placing them on the unit sphere. Notice that Gaussianity has not been used so far. Every random vector becomes a point on the same sphere after centering and normalization. Step 3: Gaussianity allows the derivation of the exact distributionThe standard multivariate normal distribution is rotationally invariant. This means that the probability density of a random vector depends only on its distance from the origin, not on its direction. Conditional on a radius, directions are uniformly distributed on a sphere. Therefore, after normalization to unit length, the vectors become uniformly distributed on the unit sphere. Assuming the vectors are independent and Gaussian, the exact sampling distribution of the correlation C can be derived by fixing one direction and measuring the surface area of all points on the sphere that produce a given correlation value. Rotational invariance guarantees that a result obtained for one fixed direction applies equally to every direction. After obtaining the result and normalizing it to 1, the exact sampling distribution of the correlation iswhereBis the Beta function. The numerator reflects the geometry of the sphere, while the denominator is the normalizing constant. Since the variance decreases as 1n−1\frac{1}{n−1}, the typical magnitude of the correlation decreases as (n)−1(\sqrt{n})^{-1}. So under the independent Gaussian null the typical correlations are of orderEquivalently, the squared correlation under the independent Gaussian model follows a Beta distributionwhich makes exact quantiles and significance thresholds easy to compute. n = 3: correlation distribution is U-shapedThis distribution is a shifted and rescaled version of the classical arcsine distribution. Histogram: empirical sampling distribution from simulated Gaussian vectorsFor a study with only 3 subjects, the two-sided 5% rejection threshold for the null hypothesis of zero population correlation is ∣r∣>0. 997. Smaller correlations are not unusual enough to reject the hypothesis of no relationship. Notice that with only 3 subjects, it is more likely to observe a large correlation than a correlation around 0, even though the variables are unrelated. n = 4: correlation follows a uniform distributionBecause the numerator in the exact sampling distribution formula simplifies to 1 whenn = 4Histogram: empirical sampling distribution from simulated Gaussian vectorsFor a study with 4 subjects, the rejection threshold remains very high: ∣r∣>0. 95∣r∣>0. 95. Smaller correlations provide insufficient evidence to reject the hypothesis of no relationship. For this sample size, all correlations are equally likely to be observed. n = 5: correlation distribution is semicircle-shapedIt is a Wigner semicircle distribution. Histogram: empirical sampling distribution from simulated Gaussian vectorsForn = 6and higher the distribution becomes bell-shaped. As the sample size increases, it approaches the asymptotic normal distribution. Step 4: when n goes to infinityAsngrows, the dimension of the sphere increases and almost all pairs of random directions become nearly orthogonal. As a result, the distribution of the correlation concentrates around zero. More precisely,Therefore, for sufficiently largenWhat about non-normal sample distributions?Surprisingly, the asymptotic result also holds for non-normal data, provided the underlying distribution has finite variance. More surprisingly, the exact sampling distribution often remains an excellent approximation even when the data is non-normal and the sample size is small. Exponential distributionThe derivation of the exact sampling distribution relies on directions being uniformly distributed on the unit sphere and therefore requires rotational invariance. The exponential distribution does not possess this property. This sphere is obtained by centering and normalizing independent exponential random vectors. Unlike the Gaussian case, the points are not uniformly distributed on the sphere and some directions are preferred. The correlation distribution no longer follows the exact law derived earlier. Histogram: empirical sampling distribution from simulated exponential vectorsFor smalln, the empirical sampling distribution of the correlation is very different from the exact sampling distribution. For this simulated exponential example, empirical quantiles would provide a more appropriate reference for hypothesis testing than either the exact Gaussian distribution or the asymptotic normal approximation. Asnincreases, the empirical sampling distribution of the correlation starts resembling the asymptotic normal distribution. Histogram: empirical sampling distribution from simulated exponential vectorsForn = 10, the empirical sampling distribution is still neither close to the asymptotic normal distribution nor to the exact sampling distribution of the correlation. The bell shape is visibly skewed. This has practical consequences: the rejection region based on the exact distribution is no longer correct, even though the observed Type I error rate may appear approximately correct or even slightly conservative. For a sample size of 100, the empirical distribution is fairly close to the asymptotic normal distribution and nearly symmetric. Histogram: empirical sampling distribution from simulated exponential vectorsAlthough some residual skewness remains, the approximation can be considered good enough. Symmetric distributionsA very interesting thing happens for simulated symmetric distributions. As long as the variance is finite, the empirical distribution is very close to the exact distribution derived under the Gaussian independence assumption. This is true for sample sizes as small asn = 3for the Laplace and uniform distributions. Histogram: empirical sampling distribution from simulated Laplace vectorsA transformed U-shaped Beta(0. 2,0. 2)\mathrm{Beta}(0. 2, 0. 2) distribution, symmetric around 0, behaves differently. Its empirical sampling distribution of Pearson’s correlation differs from the exact one. Histogram: empirical sampling distribution from simulated Beta vectorsNo wonder. The vectors are not uniformly distributed on the unit sphere. Surprisingly, it still converges to the exact sampling distribution of the correlation. Histogram: empirical sampling distribution from simulated beta vectorsFornas small as 8, the empirical distribution looks very similar to the exact one, even though the latter was derived under the assumption of rotational invariance. In these simulations, skewness appears to be the dominant source of deviation from the exact sampling distribution derived under the normality assumption. The Cauchy distribution, which has infinite variance, does not show the same convergence. But the simulated symmetric non-normal distributions with finite variance do move toward the exact sampling distribution of the correlation. Notice that the convergence happens long before the asymptotic normal approximation becomes accurate. What does it all mean for a practitioner?Small studies are susceptible to spurious correlationsA common misunderstanding is that spurious correlations only arise in large (high-dimensional) datasets. But in a small study with the same number of subjects, each sample correlation comes from exactly the same sampling distribution. It’s like rolling three fair dice. The probability of getting a total greater than 16 is close to 2%. This extreme result can occur on the first try. But as the number of rolls increases, the chance of observing at least one such outcome also increases. If you roll the dice 100 times, the probability of observing at least one sum greater than 16 is about 84%. This is why high-dimensional datasets contain extreme correlations more often, even when the variables are unrelated. But a small study with the same number of subjects draws its sample correlations from exactly the same sampling distribution. The same kind of extreme correlation can therefore occur purely by chance. Correlations should never be interpreted in isolation from the sample sizeIn many applied fields, correlations of 0. 4 or greater are often described as strong. Consider a study with only 3 subjects. Even when the variables have no true relationship, correlations are more likely to be close to ±1 than to 0. The probability that |r|>0. 4|r| > 0. 4 is close to 74% when the variables are independent. The regions of the histogram outside the dashed lines correspond to ∣r∣>0. 4|r| > 0. 4. With 10 observations, there is still a 25% chance of getting a value |r|>0. 4|r| > 0. 4. The regions of the histogram outside the dashed lines correspond to ∣r∣>0. 4|r| > 0. 4. This value cannot be considered evidence of a relationship in small-sample studies. In a study of 10 mice measuring the expression levels of about 20,000 genes against one unrelated outcome, we expect over 1300 pairs with |r|>0. 6|r| > 0. 6 even when no biological relationship exists. This is why multiple-testing corrections are routinely used in high-dimensional datasets. Does it mean small-sample studies shouldn’t be done or published? Not at all. The experiment can be replicated, and eventually, when several laboratories repeat it, a meta-analysis can place each result in context. For n = 100 subjects, the picture is different. The probability that ∣r∣>0. 4∣r∣>0. 4 is only 0. 00359% (about 36 per million). The regions of the histogram outside the dashed lines correspond to ∣r∣>0. 4|r| > 0. 4. For this sample size, absolute correlations of 0. 2 and larger are considered statistically significant. The upper 2. 5% critical value for various sample sizes is shown in the table below. Take note, however, that statistical significance does not guarantee that a relationship is real. The null distribution has support on the entire interval [−1,1][-1,1], so every possible sample correlation can arise even when the true population correlation is zero. Correlations in the rejection region are simply improbable under the null, not impossible. They are the source of type I errors in hypothesis testing and can still lead to false discoveries. The test based on the exact distribution is surprisingly robustMany software implementations, includingscipy. stats. pearsonr, use the exact null distribution of Pearson’s correlation coefficient. Because it is derived for independent Gaussian variables, researchers might think they should use other methods for non-normal data. It appears that the exact sampling distribution of Pearson’s correlation coefficient provides a surprisingly good approximation, long before the sampling distribution approaches its large-sample normal limit—sometimes even forn = 3orn = 4. This empirical robustness is remarkably strong for simulated symmetric distributions with finite variance and holds in cases far from asymptotic conditions. Theoretical explanations or references from readers are welcome. When the underlying distribution is known to be skewed, one can estimate empirical null quantiles by simulating the distribution of the correlation under the null hypothesis of independence. For the simulated exponential distribution, the estimated quantiles are:These are Monte Carlo estimates from simulations (one run); more stable values can be obtained with more simulation runs. For larger sample sizes, the midpoint moves toward 0, and the empirical quantiles move closer to both the exact Gaussian null distribution and the asymptotic normal approximation. The one takeawayIf there is only one sentence I could recommend you remember, it is this:investigate the sample size of any studythat boasts a remarkable result. It could be a genuine discovery, but it could also be two vectors that happened to land unusually close together on a sphere by chance. Feedback welcomeIf you’d like to replicate the simulations, please visit theGitHubrepository. All figures were created by me, and the simulation-based figures can be reproduced from the code there. If you enjoyed the article or have a suggestion, don’t hesitate to contact me on LinkedIn. All thoughtful feedback is greatly appreciated.