Correlation
Correlation measures how strongly two numeric variables move together. It is one number between −1 and +1, and it is a quick way to spot relationships worth investigating, as long as you remember that correlation alone never proves that one thing causes another.
Positive, negative and none
| Correlation (r) | Meaning | Example |
|---|---|---|
| Close to +1 | As one goes up, the other goes up | Hours studied and exam score |
| Close to −1 | As one goes up, the other goes down | Price of a product and units sold |
| Close to 0 | No straight-line relationship | Shoe size and exam score |
A rough guide to strength, using the size of r and ignoring the sign: below 0.2 very weak, 0.2–0.4 weak, 0.4–0.6 moderate, 0.6–0.8 strong, above 0.8 very strong. These cut-offs are only a rule of thumb; what counts as "strong" depends on the field.
Pearson's correlation coefficient
The most common measure is Pearson's r, which measures the strength of a straight-line relationship:
r = Σ(x − x̄)(y − ȳ) ÷ √[ Σ(x − x̄)² × Σ(y − ȳ)² ]
The top of the fraction is large and positive when x and y are above their means at the same time (and below them at the same time). The bottom scales the answer so it always lands between −1 and +1, whatever the units.
A worked example
A kiosk owner records the daily high temperature and the number of sodas sold on six days. The means are x̄ = 26 °C and ȳ = 140 sodas.
| Temp x | Sodas y | x − x̄ | y − ȳ | (x − x̄)(y − ȳ) | (x − x̄)² | (y − ȳ)² |
|---|---|---|---|---|---|---|
| 20 | 110 | −6 | −30 | 180 | 36 | 900 |
| 22 | 125 | −4 | −15 | 60 | 16 | 225 |
| 25 | 120 | −1 | −20 | 20 | 1 | 400 |
| 27 | 150 | 1 | 10 | 10 | 1 | 100 |
| 30 | 160 | 4 | 20 | 80 | 16 | 400 |
| 32 | 175 | 6 | 35 | 210 | 36 | 1225 |
| Totals | 560 | 106 | 3250 |
r = 560 ÷ √(106 × 3250) = 560 ÷ √344,500 ≈ 560 ÷ 586.94 ≈ 0.954, a very strong positive correlation. pandas agrees:
import pandas as pd
weather = pd.DataFrame({
"temp_c": [20, 22, 25, 27, 30, 32],
"sodas_sold": [110, 125, 120, 150, 160, 175],
})
print(round(weather["temp_c"].corr(weather["sodas_sold"]), 3))
print(weather.corr())
df.corr() returns a correlation matrix: every column against every other. The diagonal is always 1 (each column with itself), and the matrix is symmetrical. With many columns, show it as a seaborn heatmap. In pandas 2.x, add numeric_only=True if the DataFrame also contains text columns.
Is the correlation real? pearsonr
With only six days, could a correlation this strong appear by luck? scipy.stats.pearsonr returns r together with a p-value.
from scipy.stats import pearsonr
r, p = pearsonr(weather["temp_c"], weather["sodas_sold"])
print(f"r = {r:.3f}, p = {p:.4f}")
The p-value answers: if there were really no linear relationship between temperature and soda sales, how often would six random days produce a correlation at least this strong (in either direction)? The answer is about 0.3% of the time, which is rare, so the relationship is unlikely to be pure chance. It does not tell you the relationship is causal or how big it is in the wider world; with six points the true correlation could be noticeably weaker. The p-value also assumes the data is roughly normally distributed. Hypothesis Testing explains p-values in detail.
Spearman's rank correlation
Spearman's ρ (rho) ranks each variable from smallest to largest and correlates the ranks. It measures whether the relationship is consistently upward or downward (monotonic), even if it isn't a straight line, and it is much less affected by outliers.
print(round(weather["temp_c"].corr(weather["sodas_sold"], method="spearman"), 3))
Temperatures rank 1 to 6 in order. Soda sales rank 1, 3, 2, 4, 5, 6, because 120 on the 25 °C day is lower than 125 on the 22 °C day. That single swap lowers ρ slightly from a perfect 1. scipy.stats.spearmanr gives the same value with a p-value.
Pitfalls
Correlation is not causation
Umbrella sales and matatu delays in Nairobi rise and fall together, but umbrellas don't cause traffic. Both are driven by a third factor, rain, called a confounder. Even when two things really are connected, correlation can't tell you which causes which. To show cause and effect you need a controlled experiment or careful causal analysis.
Non-linear relationships
Pearson's r only detects straight-line patterns. A perfect U-shape can score zero:
x = pd.Series([-2, -1, 0, 1, 2]) y = x ** 2 # 4, 1, 0, 1, 4: completely determined by x print(round(x.corr(y), 3))
y depends entirely on x, yet r is 0. Always draw a scatter plot before trusting a correlation.
Outliers and small samples
One extreme point can create a strong correlation where there is none, or hide a real one. With few data points, r can swing a lot from sample to sample. Check a scatter plot, try Spearman, and report the sample size.
Reading r as a slope or a percentage. r = 0.5 does not mean "half of the sales are explained by temperature" or "sales rise 0.5 per degree". It only describes how tightly the points cluster around a straight line. For the size of the effect, fit a line with linear regression.
Key points
- Correlation (r) runs from −1 to +1 and measures how closely two variables follow a straight line.
- Use
Series.corr()orDataFrame.corr(); usescipy.stats.pearsonrfor a p-value. - Spearman's ρ works on ranks, catches monotonic curves and resists outliers.
- Correlation is not causation; always plot the data and watch for confounders, outliers and curves.
Tip: use the ← and → keys to move between pages.
