Skip to content
Kuza Ndoto Academy

Linear Regression

IntermediateChapter 3 · Page 9 of 264 min readUpdated 15 Nov 2026

Linear regression predicts a number by fitting a straight line (or, with many features, a flat plane) through the data. It's the simplest regression algorithm, very fast, easy to explain, and the foundation for many more advanced models.

The equation of a line

With one feature, the model is the line you met in school maths:

y = m x + c

  • m is the slope (in ML, the coefficient or weight): how much y changes when x increases by 1.
  • c is the intercept (the bias): the value of y when x is 0.

With several features it becomes y = w₁x₁ + w₂x₂ + … + b, one weight per feature. Training means finding the weights that make predictions as close as possible to the real labels.

What is the "best" line?

For each point, the residual is the actual value minus the predicted value. Linear regression chooses the line that minimises the sum of squared residuals (ordinary least squares). Squaring makes all errors positive and punishes big misses more than small ones. Averaging the squared residuals gives the mean squared error (MSE).

A worked example by hand

Years of experience vs. monthly salary (in thousands of KES) for five junior developers:

x (years)12345
y (KES '000)3035455060

For one feature, least squares has a neat formula: m = Σ(x − x̄)(y − ȳ) / Σ(x − x̄)², and c = ȳ − m x̄.

  1. Means: x̄ = 15 / 5 = 3 and ȳ = 220 / 5 = 44.
  2. Deviations: x − 3 = −2, −1, 0, 1, 2 and y − 44 = −14, −9, 1, 6, 16.
  3. Sum of products: 28 + 9 + 0 + 6 + 32 = 75. Sum of squared x deviations: 4 + 1 + 0 + 1 + 4 = 10.
  4. Slope m = 75 / 10 = 7.5. Intercept c = 44 − 7.5 × 3 = 21.5.

So salary ≈ 7.5 × years + 21.5: each extra year of experience is worth about KES 7,500 a month. The predictions for x = 1 to 5 are 29, 36.5, 44, 51.5 and 59, giving residuals 1, −1.5, 1, −1.5 and 1. The squared residuals add up to 7.5, so MSE = 7.5 / 5 = 1.5.

12345 2030405060 Years of experience Salary (KES '000) y = 7.5x + 21.5
Figure 1. The five developers (dots), the least-squares line (gold) and the residuals (red), which are 1, -1.5, 1, -1.5 and 1 thousand KES. Scale: 90 px per year, 5 px per KES 1,000.
import numpy as np

x = np.array([1, 2, 3, 4, 5])
y = np.array([30, 35, 45, 50, 60])

m = np.sum((x - x.mean()) * (y - y.mean())) / np.sum((x - x.mean()) ** 2)
c = y.mean() - m * x.mean()
print(m, c)
7.5 21.5

The same model in scikit-learn

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score

X = x.reshape(-1, 1)            # scikit-learn needs a 2-D feature table
model = LinearRegression()
model.fit(X, y)

print(f"Slope: {model.coef_[0]:.2f}")
print(f"Intercept: {model.intercept_:.2f}")
print(f"6 years: {model.predict([[6]])[0]:.1f}k KES")

y_pred = model.predict(X)
print(f"MSE: {mean_squared_error(y, y_pred):.2f}")
print(f"R^2: {r2_score(y, y_pred):.3f}")
Slope: 7.50 Intercept: 21.50 6 years: 66.5k KES MSE: 1.50 R^2: 0.987

R² (the coefficient of determination) compares your errors with those of a "model" that always predicts the mean. Here the total squared deviation from the mean is 196 + 81 + 1 + 36 + 256 = 570, so R² = 1 − 7.5 / 570 ≈ 0.987. An R² of 1 is a perfect fit; 0 is no better than guessing the average.

Gradient descent: how bigger models learn

The formula above works for simple cases, but many models are trained by gradient descent instead. Start with any slope and intercept, measure the MSE, then nudge each value a small step in the direction that reduces the error the fastest (downhill on the error surface). Repeat thousands of times. The step size is the learning rate: too big and you overshoot; too small and training crawls. The same idea trains neural networks.

Strengths, limits and assumptions

  • Strengths: fast, needs little data, and the coefficients are easy to explain.
  • Assumes a roughly straight-line relationship. If the true pattern curves, the model underfits; try adding features such as x² or use a tree-based model.
  • Sensitive to outliers, because squared errors give extreme points a big pull on the line.
  • Highly correlated features make individual coefficients unstable. Regularised versions (Ridge, Lasso) help.
Common mistake

Extrapolating far beyond the data. The model says 30 years of experience means 7.5 × 30 + 21.5 = KES 246,500 a month, but it never saw anyone with more than 5 years. Salaries don't grow in a straight line forever. Trust predictions only within the range you trained on.

Key points

  • Linear regression fits y = m x + c (or one weight per feature) to predict numbers.
  • It minimises the sum of squared residuals; MSE and R² measure the fit.
  • Coefficients are easy to interpret: the change in y per unit change in a feature.
  • Watch out for curved relationships, outliers and extrapolation.
Up next · Supervised LearningLogistic RegressionPredict yes/no outcomes.
→

Tip: use the ← and → keys to move between pages.