Skip to content
Kuza Ndoto Academy

Hypothesis Testing

Beginner → IntermediateChapter 5 · Page 18 of 206 min readUpdated 15 Nov 2026

You notice that students who used a new revision method scored higher than those who didn't. Is the new method really better, or did the difference happen by chance? Hypothesis testing is a structured way to answer questions like this using data.

The big idea

Any sample contains random variation. Even if two methods were equally good, two groups of students would almost never get identical average scores. A hypothesis test asks: if there were no real difference, how surprising would our result be? If it would be very surprising, we have evidence of a real difference.

  • The null hypothesis (H₀) is the "nothing special is happening" claim: the two methods give the same average score.
  • The alternative hypothesis (H₁) is what we are looking for evidence of: the average scores differ.

We never prove H₁ directly. We check whether the data is hard to explain if H₀ were true.

The steps

  1. State H₀ and H₁.
  2. Choose a significance level α (alpha) before looking at the results. 0.05 is the usual choice.
  3. Collect data and choose a suitable test (here, a t-test comparing two means).
  4. Calculate the test statistic and its p-value.
  5. If p ≤ α, reject H₀ ("statistically significant"). If p > α, you fail to reject H₀.
  6. Explain the result in plain language, including the size of the difference.

What a p-value is (and isn't)

The p-value is the probability of getting a result at least as extreme as the one you observed, assuming the null hypothesis is true. A small p-value means your data would be unusual in a world where H₀ is true, which counts as evidence against H₀.

Common mistake

A p-value is not the probability that the null hypothesis is true, and 1 − p is not the probability that your idea is right. p = 0.03 does not mean "there is a 3% chance the methods are the same". It means: if the methods were the same, results at least this far apart would turn up about 3% of the time. A p-value also says nothing about how big or important a difference is.

Example: comparing two groups with a t-test

Ten students were randomly split into two groups of five. One group revised the old way; the other used a new method. Their test scores:

import numpy as np
from scipy import stats

old_method = np.array([62, 65, 70, 58, 65])
new_method = np.array([70, 74, 68, 77, 71])

print(old_method.mean(), new_method.mean())
print(old_method.var(ddof=1), new_method.var(ddof=1))

result = stats.ttest_ind(new_method, old_method)
print(f"t = {result.statistic:.3f}")
print(f"p = {result.pvalue:.4f}")
64.0 72.0 19.5 12.5 t = 3.162 p = 0.0133

The new-method group scored 8 points higher on average. ttest_ind runs an independent two-sample t-test with H₀: "the two population means are equal".

Where the numbers come from

  • Sample variances (ddof=1): old 78 ÷ 4 = 19.5, new 50 ÷ 4 = 12.5.
  • Pooled variance: (78 + 50) ÷ (5 + 5 − 2) = 128 ÷ 8 = 16, so the pooled standard deviation is 4.
  • Standard error of the difference: 4 × √(1/5 + 1/5) ≈ 2.530.
  • t = difference ÷ standard error = 8 ÷ 2.530 ≈ 3.162, with 8 degrees of freedom.

The t value says the difference is about 3.2 standard errors away from zero. If the two methods were truly equal, a gap this large (in either direction) would appear only about 1.3% of the time. Since p = 0.0133 is below α = 0.05, we reject H₀: the data gives evidence that the new method leads to higher scores. Because students were assigned randomly, it is reasonable to credit the method itself; with only ten students, a larger follow-up study would make the conclusion much firmer.

observed t = 3.16 −4 −3 −2 −1 0 1 2 3 4 −2.306 2.306 rejection region, α = 0.05 p-value: both tails ≈ 0.013 t-distribution, 8 df (illustration)
Figure 1. If H₀ were true, t would follow this curve. Values beyond ±2.306 (shaded) happen only 5% of the time in total. The observed t = 3.16 lands in that region; the p-value is the tiny area beyond ±3.16 in both tails.
Note

By default ttest_ind assumes both groups have the same variance (Student's t-test). Many statisticians prefer Welch's t-test, stats.ttest_ind(new_method, old_method, equal_var=False), which doesn't make that assumption. Here it gives the same t (because the groups are the same size) but slightly fewer effective degrees of freedom (about 7.6), so the p-value is slightly larger, roughly 0.014. The conclusion doesn't change. If you only care whether the new method is better (not just different), and decided that before seeing the data, add alternative="greater" for a one-sided test; here that halves the p-value to about 0.0067.

Two kinds of error

H₀ is actually trueH₀ is actually false
Reject H₀Type I error (false positive)Correct decision
Fail to reject H₀Correct decisionType II error (false negative)

α is the Type I error rate you accept: with α = 0.05, about 1 in 20 tests of a true null hypothesis will wrongly come out "significant". Small samples make Type II errors more likely, because real differences get lost in the noise. The probability of detecting a real effect is called the test's power.

Statistical vs. practical significance

"Significant" in statistics only means "unlikely to be chance alone". With huge samples, a tiny, useless difference can be significant; with small samples, an important difference can fail to reach significance. Always report the effect size too. A common one for two means is Cohen's d = difference ÷ pooled standard deviation. Here d = 8 ÷ 4 = 2.0, a very large effect by usual standards, and 8 marks is a difference a teacher would care about.

Common mistake

"We failed to reject H₀, so the methods are the same." No: a non-significant result means the data didn't provide enough evidence, not that there is no difference. Another trap is p-hacking: running many tests, or trying different subsets of the data, until something gives p < 0.05. If you run 20 tests where nothing is going on, you should expect about one false positive.

Choosing a test

QuestionTestSciPy function
Do two independent groups have different means?Two-sample t-teststats.ttest_ind()
Did the same people change (before vs. after)?Paired t-teststats.ttest_rel()
Does one group's mean differ from a target value?One-sample t-teststats.ttest_1samp()
Are two categorical variables related?Chi-square teststats.chi2_contingency()
Are two numeric variables correlated?Correlation teststats.pearsonr() (see Correlation)

Key points

  • H₀ says "no effect"; a test checks how surprising the data would be if H₀ were true.
  • The p-value is the probability of a result at least this extreme assuming H₀ is true, not the probability that H₀ is true.
  • Reject H₀ when p ≤ α (usually 0.05); otherwise you fail to reject it, which is not proof of no effect.
  • Report the effect size and sample size, not just the p-value.
Up next · New chapter: ProjectsProject: Sales AnalysisAnalyse a real sales dataset end to end.
→

Tip: use the ← and → keys to move between pages.