Skip to content
Kuza Ndoto Academy

Neural Networks

IntermediateChapter 6 · Page 19 of 265 min readUpdated 15 Nov 2026

A neural network is a model built from many simple units, called artificial neurons, arranged in layers. Each neuron does a tiny calculation, but stacked together and trained on lots of data, networks can recognise faces, transcribe speech and power chatbots. They are the foundation of deep learning.

One artificial neuron

A neuron takes several inputs, multiplies each by a weight, adds them up with a bias, and passes the result through an activation function:

output = activation(w₁x₁ + w₂x₂ + … + b)

The weights say how important each input is; the bias shifts the result up or down. Without the activation, this is just linear regression. With a sigmoid activation, a single neuron is logistic regression.

Worked example: inputs x = (2, 3), weights w = (0.5, 0.25), bias b = −1. Then z = 2 × 0.5 + 3 × 0.25 − 1 = 1 + 0.75 − 1 = 0.75.

import numpy as np

x = np.array([2.0, 3.0])       # inputs
w = np.array([0.5, 0.25])      # weights
b = -1.0                       # bias

z = np.dot(x, w) + b
print("z =", z)
print("ReLU:", max(0.0, z))
print("sigmoid:", round(1 / (1 + np.exp(-z)), 3))
z = 0.75 ReLU: 0.75 sigmoid: 0.679
x1 = 2 x2 = 3 w1 = 0.5 w2 = 0.25 b = -1 Σ + b z = 0.75 ReLU max(0, z) 0.75 output z = 2(0.5) + 3(0.25) - 1 = 0.75
Figure 1. The neuron from the code: weighted sum plus bias gives z = 0.75, and ReLU passes it through unchanged because it is positive.

Activation functions

FunctionFormulaTypical use
ReLUmax(0, z)Hidden layers (the default choice)
Sigmoid1 / (1 + e−z), output 0 to 1Output layer for yes/no problems
SoftmaxTurns a list of scores into probabilities that sum to 1Output layer for multi-class problems
TanhOutput −1 to 1Some hidden layers, older recurrent networks

Activations must be non-linear. Stacking linear layers without them just gives another linear function, however many layers you add. Non-linearity is what lets networks learn curves and complex patterns.

Layers

  • Input layer: one value per feature (for a 28 × 28 pixel image, 784 values).
  • Hidden layers: neurons that each receive every output of the previous layer (a dense or fully connected layer). Early layers learn simple patterns; later ones combine them into complex ones. "Deep" learning simply means many hidden layers.
  • Output layer: one neuron for regression, one sigmoid neuron for binary classification, or one softmax neuron per class.

A network with 4 inputs, a hidden layer of 16 neurons and 3 outputs has (4 × 16 + 16) + (16 × 3 + 3) = 80 + 51 = 131 trainable weights and biases.

Input layer (4) Hidden layer (16, 5 shown) Output layer (3, softmax) sepal lengthsepal widthpetal lengthpetal width setosaversicolorvirginica
Figure 2. The 4-16-3 network used in the examples below. Each line is a weight; every neuron also has a bias, giving 131 parameters in total.

How networks learn: loss, gradients and backpropagation

  1. Forward pass: feed a batch of examples through the network to get predictions.
  2. Loss: measure how wrong they are (mean squared error for numbers, cross-entropy for classes).
  3. Backpropagation: working backwards from the output, use the chain rule of calculus to compute the gradient of the loss with respect to every weight, meaning how much the loss would change if that weight changed slightly.
  4. Update: nudge every weight a small step against its gradient (gradient descent). The step size is the learning rate.

One pass through the whole training set is an epoch; training usually runs for many epochs. Here is gradient descent on a single weight. We want w × 2 to equal 4 (so the right answer is w = 2), starting from w = 1, with loss (w × x − y)² and gradient 2(w × x − y) × x:

w, x, y, lr = 1.0, 2.0, 4.0, 0.1

for step in range(3):
    pred = w * x
    loss = (pred - y) ** 2
    grad = 2 * (pred - y) * x
    w = w - lr * grad
    print(step, round(loss, 4), round(w, 4))
0 4.0 1.8 1 0.16 1.96 2 0.0064 1.992

At step 0 the prediction is 2, the loss is 4 and the gradient is 2 × (2 − 4) × 2 = −8, so w moves to 1 − 0.1 × (−8) = 1.8. The loss shrinks each step as w approaches 2. Backpropagation does exactly this for millions of weights at once.

A network in scikit-learn

scikit-learn includes a simple neural network, MLPClassifier (multi-layer perceptron), which is fine for small tabular problems:

from sklearn.datasets import load_iris
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

X, y = load_iris(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y)

net = make_pipeline(StandardScaler(),
                    MLPClassifier(hidden_layer_sizes=(16,), max_iter=2000, random_state=42))
net.fit(X_train, y_train)
print(f"Test accuracy: {net.score(X_test, y_test):.2f}")

You should see an accuracy of roughly 0.93 to 1.00. Neural networks need scaled inputs to train well.

The same idea in Keras

For real deep learning, people use frameworks such as TensorFlow/Keras or PyTorch, which run on GPUs. This example needs an extra installation (pip install tensorflow, which includes Keras 3) and reuses the scaled iris data:

import keras
from keras import layers
from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_s = scaler.fit_transform(X_train)
X_test_s = scaler.transform(X_test)

model = keras.Sequential([
    keras.Input(shape=(4,)),
    layers.Dense(16, activation="relu"),
    layers.Dense(3, activation="softmax"),
])
model.compile(optimizer="adam",
              loss="sparse_categorical_crossentropy",
              metrics=["accuracy"])
model.fit(X_train_s, y_train, epochs=100, verbose=0)

loss, accuracy = model.evaluate(X_test_s, y_test, verbose=0)
print(f"Test accuracy: {accuracy:.2f}")

model.summary() would report 131 trainable parameters, matching our hand count. Keras starts from random weights, so results vary slightly between runs; expect accuracy above 0.9.

Common mistake

Using a neural network for every problem. On small tabular datasets, networks are often beaten by random forests or gradient boosting, which train faster and need less tuning. Networks shine with large amounts of images, audio or text.

Key points

  • A neuron computes a weighted sum plus a bias, then applies a non-linear activation.
  • Layers of neurons learn increasingly complex patterns.
  • Training repeats: forward pass, loss, backpropagation, gradient descent update.
  • Scale inputs, and choose networks when you have lots of unstructured data.
Up next · Deep Learning & GenAIConvolutional Neural NetworksHow computers see images.
→

Tip: use the ← and → keys to move between pages.