Skip to content
Kuza Ndoto Academy

How Large Language Models Work

IntermediateChapter 6 · Page 21 of 265 min readUpdated 15 Nov 2026

A large language model (LLM) is a neural network trained on enormous amounts of text to do one thing extremely well: predict the next piece of text. Chatbots such as ChatGPT, Claude and Gemini are built on LLMs. Understanding how they work helps you use them well and recognise their limits.

The core idea: next-token prediction

Given the text so far, an LLM outputs a probability for every possible next token. It picks one, adds it to the text, and repeats. Every answer, essay or piece of code an LLM writes is produced this way, one token at a time.

The simplest possible "language model" just counts which word follows which. Here is one trained on three tiny sentences:

from collections import Counter, defaultdict

text = "I love Nairobi . I love Kampala . I visit Nairobi often ."
words = text.split()

following = defaultdict(Counter)
for current, nxt in zip(words, words[1:]):
    following[current][nxt] += 1

print(following["I"].most_common())
print(following["love"].most_common())
[('love', 2), ('visit', 1)] [('Nairobi', 1), ('Kampala', 1)]

After "I", this model predicts "love" with probability 2/3 and "visit" with 1/3. A real LLM does the same job, but instead of looking at one previous word it considers thousands of previous tokens at once, and instead of counting it uses a neural network with billions of learned weights. That's what lets it follow grammar, facts, style and long arguments.

Step 1: tokens

LLMs don't read letters or whole words; they read tokens, which are common chunks of text. A frequent word like "the" is usually one token; a rarer word may be split into pieces, such as "un" + "believ" + "able". Each token has an ID number. As a rough guide, one token is about three-quarters of an English word, but this varies by model and language. Text in languages that were less common in the training data, including Kiswahili, often needs more tokens per word. Limits such as the context window (how much text the model can consider at once) and API prices are measured in tokens.

Step 2: embeddings

Each token ID is converted into an embedding: a long list of numbers (a vector, often thousands of numbers long) that represents its meaning. These vectors are learned during training. Tokens used in similar ways end up with similar vectors, so "Nairobi" sits near "Kampala" and "Dodoma" in this space. Information about each token's position in the sequence is added too, since word order matters.

Step 3: transformers and attention

Modern LLMs use the transformer architecture, introduced by Google researchers in 2017. Its key ingredient is self-attention: for every token, the model works out how much attention to pay to every earlier token when building its understanding.

Consider "She deposited the cash at the bank" versus "They fished from the river bank". Attention lets the vector for "bank" absorb information from "cash" in the first sentence and "river" in the second, so the model can tell the two meanings apart. In practice, each token produces a query, a key and a value vector. Comparing a token's query with every key gives attention scores, a softmax turns them into weights, and the token's new representation is the weighted mix of the values. A transformer stacks dozens of these layers, each with many attention "heads" running in parallel.

Step 4: choosing the next token

The final layer produces a score (a logit) for every token in the vocabulary, and a softmax turns the scores into probabilities. The temperature setting reshapes those probabilities before a token is sampled. Suppose the prompt is "The capital city of Kenya is" and three candidates have logits 4, 2 and 1:

import numpy as np

candidates = ["Nairobi", "Mombasa", "Kisumu"]
logits = np.array([4.0, 2.0, 1.0])

def softmax(z, temperature=1.0):
    e = np.exp(z / temperature)
    return e / e.sum()

for t in [1.0, 2.0]:
    probs = softmax(logits, t)
    print(t, [f"{p:.3f}" for p in probs])
1.0 ['0.844', '0.114', '0.042'] 2.0 ['0.629', '0.231', '0.140']

At temperature 1, "Nairobi" gets 84% of the probability. At temperature 2 the distribution flattens and less likely tokens get picked more often. Low temperature gives focused, repeatable answers; higher temperature gives more varied, creative ones, with more risk of errors.

Text The capital city of Kenya is Tokens The capital city of Kenya is Embeddings [0.12, -0.48, …] [0.91, 0.05, …] vector per token Transformer attention attention attention Next token Nairobi 0.844 Mombasa 0.114 Kisumu 0.042 the chosen token ("Nairobi") is appended, then the model predicts again
Figure 1. One step of generation. The token split and embedding numbers are illustrative; the probabilities are the temperature-1 values from the code (bar length = probability, 80 px = 1.0).

How LLMs are trained

  1. Pre-training: the model reads a huge collection of text (web pages, books, code) and learns to predict the next token. This is self-supervised: the text itself supplies the answers. It takes enormous computing power and produces a base model that can continue text but doesn't reliably follow instructions.
  2. Fine-tuning: the base model is trained further on smaller, carefully prepared datasets, such as examples of good responses to instructions (instruction tuning).
  3. Preference training: people (and sometimes other models) compare responses, and the model is trained to prefer helpful, honest and harmless ones, for example with reinforcement learning from human feedback (RLHF).

Organisations can also fine-tune an existing model on their own examples to adapt its style or specialise it. To give a model knowledge it didn't learn in training, such as your company's policies, retrieval-augmented generation is usually cheaper and easier to keep up to date.

Limitations you must know

  • Hallucination: LLMs can produce fluent, confident statements that are false, including made-up references, statistics and quotes. They generate likely-sounding text; they don't look facts up unless connected to a tool that does.
  • Knowledge cut-off: a model knows little about events after its training data was collected, unless given search or documents.
  • Bias: models absorb biases and gaps in their training text, which may under-represent African languages and perspectives.
  • Inconsistency: rephrasing a question can change the answer.
Warning

Never paste an LLM's answer into an assignment, report or medical, legal or financial decision without checking it. Verify facts, figures and citations against reliable sources, and don't share confidential or personal data with tools whose data policies you haven't checked.

Key points

  • An LLM repeatedly predicts the next token, using probabilities from a neural network.
  • Text becomes tokens, tokens become embeddings, and transformer layers use attention to model context.
  • Pre-training teaches language and knowledge; fine-tuning and preference training make the model follow instructions.
  • LLMs can hallucinate, so check important outputs.
Up next · Deep Learning & GenAIPrompt EngineeringGet reliable results from AI models.
→

Tip: use the ← and → keys to move between pages.