Skip to content
Kuza Ndoto Academy

Principal Component Analysis

IntermediateChapter 4 · Page 15 of 264 min readUpdated 15 Nov 2026

Principal Component Analysis (PCA) reduces many features to a few new ones, called principal components, while keeping as much of the information (variance) as possible. It's used to visualise high-dimensional data, speed up models and remove redundancy between correlated features.

The intuition

Imagine a dataset with each student's height in centimetres and height in inches. That's two columns, but really only one piece of information. PCA finds this automatically: it rotates the axes so that the first new axis points along the direction where the data is most spread out, and the second axis captures what's left.

  • PC1 (the first principal component) is the direction of greatest variance.
  • PC2 is the direction of greatest remaining variance, at right angles to PC1.
  • And so on: there are as many components as original features, ordered from most to least important.

You keep the first few components and drop the rest, losing only a little information.

A tiny worked example

Take three points that lie exactly on a line: (1, 1), (2, 2) and (3, 3). Along the original x-axis they spread from 1 to 3, and the same on the y-axis. But along the diagonal direction (1, 1)/√2, their positions are 1.41, 2.83 and 4.24, and across the diagonal they are all 0. So PC1 is the diagonal, and it explains 100% of the variance; PC2 explains 0%. Two features shrink to one with no loss at all.

Real data is noisier, so later components keep a little variance, but the idea is the same. Mathematically, the components are the eigenvectors of the data's covariance matrix, and each one's variance is its eigenvalue; scikit-learn computes them for you.

0123456 123456 PC1 PC2 Points: (1,2) (2,1) (3,3) (4,5) (5,4) PC1 runs along y = x variance share 18 / 20 = 90% PC2 runs across it variance share 2 / 20 = 10% dashed lines show each point's projection onto PC1
Figure 1. A slightly noisier version of the example. The points have mean (3, 3); their sums of squared deviations are 10 for x and 10 for y, and the sum of cross-products is 8. That puts PC1 on the diagonal, with eigenvalue 18 out of a total of 20. Keeping only PC1 keeps 90% of the variance.

PCA in scikit-learn

The iris dataset has 4 features. Let's reduce it to 2 so it can be drawn on a flat chart:

from sklearn.datasets import load_iris
from sklearn.decomposition import PCA
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X_scaled = StandardScaler().fit_transform(X)    # always scale before PCA

pca = PCA(n_components=2)
X_2d = pca.fit_transform(X_scaled)
print(X.shape, "->", X_2d.shape)
(150, 4) -> (150, 2)
print(pca.explained_variance_ratio_.round(3))
print(pca.explained_variance_ratio_.sum().round(3))

On standardised iris data you should see PC1 explaining about 73% of the variance and PC2 about 23%, so the two components together keep about 96% of the information in the original four features. Plotting X_2d[:, 0] against X_2d[:, 1], coloured by species, shows the three species clearly (see Matplotlib Basics).

How many components?

  • For visualisation, use 2 or 3.
  • For modelling, keep enough to explain a target share of variance. Passing a fraction does this for you: PCA(n_components=0.95) keeps the smallest number of components that explains at least 95%.
  • Plot the cumulative explained variance (pca.explained_variance_ratio_.cumsum()) and look for where it levels off.

Using PCA inside a model

from sklearn.pipeline import make_pipeline
from sklearn.linear_model import LogisticRegression

model = make_pipeline(StandardScaler(), PCA(n_components=0.95), LogisticRegression())
# model.fit(X_train, y_train) learns the scaling, the components and the classifier
# from the training data only
Common mistake

Forgetting to scale. PCA looks for directions of largest variance, so a feature measured in big units (income in KES) will dominate PC1 just because its numbers are larger. Standardise first. And as with every preprocessing step, fit PCA on the training data only, ideally in a pipeline, to avoid leakage.

Strengths and limits

  • Strengths: fast, removes redundancy between correlated features, reduces noise, helps visualisation, and can speed up slow models like KNN.
  • Limits: components are mixtures of all original features, so they're hard to explain ("PC1 = 0.52 × sepal length + …"). PCA only finds straight-line (linear) structure. And high variance doesn't always mean useful for prediction.

PCA is unsupervised: it never looks at the labels. It's often paired with k-means to visualise clusters.

Key points

  • PCA creates new uncorrelated features (components) ordered by how much variance they capture.
  • explained_variance_ratio_ tells you how much information each component keeps.
  • Standardise first, and fit on training data only.
  • Components are compact but hard to interpret.
Up next · New chapter: Model EvaluationEvaluation MetricsAccuracy, precision, recall and more.
→

Tip: use the ← and → keys to move between pages.