Descriptive Statistics
Descriptive statistics summarise a set of numbers in a few values: where the centre is, how spread out the data is and whether anything looks unusual. They are the first thing to calculate when you meet a new dataset.
Our example data
Eight students recorded how much they spent on airtime and bundles in one week (KES): 200, 400, 400, 400, 500, 500, 700 and 900.
Measures of centre
- Mean (average): add the values and divide by how many there are. 4000 ÷ 8 = 500.
- Median: the middle value once the data is sorted. With an even count, it is the average of the two middle values: (400 + 500) ÷ 2 = 450.
- Mode: the most frequent value, here 400. It is the only one of the three that works for categories too, such as the most common county.
import pandas as pd
airtime = pd.Series([200, 400, 400, 400, 500, 500, 700, 900]) # KES per week
print("Mean:", airtime.mean())
print("Median:", airtime.median())
print("Mode:", airtime.mode().tolist())
print("Range:", airtime.max() - airtime.min())
mode() returns a Series because data can have several modes, so tolist() turns it into a plain list.
Mean or median?
The mean uses every value, so a single extreme value pulls it a long way. The median only cares about the middle, so it resists extremes. Look at the monthly pay of six people at a small business, where one is the owner:
salaries = pd.Series([25000, 28000, 30000, 32000, 35000, 400000])
print("Mean:", round(salaries.mean(), 2))
print("Median:", salaries.median())
Saying "the average worker here earns over KES 91,000" would be misleading: five of the six people earn KES 35,000 or less. For skewed data such as incomes, house prices, order values or M-Pesa transaction amounts, report the median (or both). Use the mean when the data is roughly symmetrical with no extreme values.
Measures of spread
Two datasets can share a mean and still be very different. Spread tells you how far values typically are from the centre.
- Range = maximum − minimum. Simple, but decided entirely by the two most extreme values.
- Variance: the average squared distance from the mean.
- Standard deviation: the square root of the variance, back in the original units (KES).
Population vs. sample standard deviation
If your data is the whole group you care about (the population), divide by N. If it is a sample used to estimate a bigger population, divide by n − 1. Dividing by n − 1 (called Bessel's correction) corrects the tendency of a sample to underestimate the true spread.
| Population | Sample | |
|---|---|---|
| Variance | σ² = Σ(x − μ)² / N | s² = Σ(x − x̄)² / (n − 1) |
| Standard deviation | σ = √σ² | s = √s² |
| In code | ddof=0 | ddof=1 |
By hand for the airtime data: the distances from the mean (500) are −300, −100, −100, −100, 0, 0, 200 and 400. Their squares add up to 90,000 + 3 × 10,000 + 0 + 0 + 40,000 + 160,000 = 320,000.
- Population: 320,000 ÷ 8 = 40,000, so σ = √40,000 = 200.
- Sample: 320,000 ÷ 7 ≈ 45,714.3, so s ≈ 213.81.
import numpy as np
print("Sample variance:", round(airtime.var(), 1))
print("Sample std (pandas default, ddof=1):", round(airtime.std(), 2))
print("Population std (ddof=0):", round(airtime.std(ddof=0), 2))
print("NumPy default (ddof=0):", round(np.std(airtime.to_numpy()), 2))
print("NumPy sample (ddof=1):", round(np.std(airtime.to_numpy(), ddof=1), 2))
pandas and NumPy use different defaults. Series.std() and DataFrame.std() use ddof=1 (sample), while np.std() uses ddof=0 (population). The same numbers can therefore give two different answers. Decide which one you need and always pass ddof explicitly when it matters.
Quartiles and the interquartile range
Quartiles split sorted data into four equal parts. Q1 (the 25th percentile) has a quarter of the values below it, Q2 is the median, and Q3 (the 75th percentile) has three-quarters below it. The interquartile range (IQR = Q3 − Q1) is the spread of the middle half of the data, and like the median it ignores extremes.
print(airtime.quantile([0.25, 0.5, 0.75]))
pandas finds quartiles by linear interpolation: for Q3 it looks 75% of the way along the sorted list, which falls a quarter of the way between 500 and 700, giving 550. Other tools may use slightly different methods, so small differences are normal.
Everything at once with describe()
print(airtime.describe())
describe() reports the count, mean, sample standard deviation, minimum, quartiles and maximum. On a DataFrame it does this for every numeric column at once.
Finding outliers
An outlier is a value far from the rest. A common rule (used by box plots) flags anything more than 1.5 × IQR below Q1 or above Q3.
q1 = airtime.quantile(0.25)
q3 = airtime.quantile(0.75)
iqr = q3 - q1
lower = q1 - 1.5 * iqr
upper = q3 + 1.5 * iqr
print("IQR:", iqr)
print("Fences:", lower, upper)
print(airtime[(airtime < lower) | (airtime > upper)])
The student who spent KES 900 is flagged. An outlier is a signal to investigate, not an automatic reason to delete: it may be a typing error (90 entered as 900) or a genuine heavy user. Decide based on what you find, and say what you did.
The shape of the data
Comparing the mean and median hints at the shape. When they are close, the data is roughly symmetrical. When the mean is noticeably above the median, a long tail of large values is pulling it up (right-skewed, typical of incomes and order sizes). When the mean is below the median, the data is left-skewed. Here the mean (500) is above the median (450), a mild right skew caused by the 700 and 900 values. A histogram or box plot (see Seaborn) shows the shape directly.
Key points
- Centre: mean, median and mode. Prefer the median for skewed data or data with extremes.
- Spread: range, variance, standard deviation and IQR.
- Sample statistics divide by n − 1 (
ddof=1, pandas' default); population statistics divide by N (ddof=0, NumPy's default). - Use
describe()for a quick summary and the 1.5 × IQR rule to flag outliers for investigation.
Tip: use the ← and → keys to move between pages.
