There is no small expert behind the chat box, and no library of finished answers waiting to be retrieved. Underneath is a very large collection of numbers performing the same basic move again and again.
This is worth understanding for a practical reason, not an academic one. Once you can see the mechanism, the model's characteristic failures stop being mysterious and become predictable — which is the difference between hoping an AI system behaves and being able to design controls around it.
So let us follow one prompt all the way through. (This describes the general architecture of GPT-style decoder models; the internal engineering of any particular commercial product is not fully public.)
Text becomes tokens
Long before you type anything, the model is trained on enormous text collections — websites, books, articles, documentation, conversations, code — filtered, cleaned and largely deduplicated. But the model cannot swallow paragraphs. Text is first converted into tokens.
A token can be a whole word, part of a word, punctuation, or a space plus nearby letters. A common word may be one token; an unusual name may be split into several. This is why models can be strangely bad at counting letters or handling rare spellings — they do not see text as you do. They see a sequence of token IDs, which are just numbers from a vocabulary.
Numbers become meaning
A token ID is only a label. Token 8,421 is not inherently wiser than token 8,420. So each token is mapped to an embedding: a long list of learned numerical features, which you can picture as a starting coordinate on an enormous map.
Those coordinates acquire structure during training. "King" ends up closer to "queen" than to "microwave". Order matters too — the dog chased the cat is not the cat chased the dog — so transformers encode position as well as identity, using positional embeddings or rotary position information.
Crucially, these representations become contextual as they move through the network. "Bank" in "river bank" develops differently from "bank" in "bank account". Meaning is not fetched from a permanent dictionary entry; it is constructed from the surrounding text.
Attention: what matters right now
The transformer contains many repeated blocks. Each lets information move between tokens, then transforms it further. The famous part is self-attention.
Take the sentence: "The trophy did not fit inside the suitcase because it was too large." What was too large? To resolve "it", the model needs earlier context. In a GPT-style causal model, each position can use itself and positions before it, but cannot peek at future tokens during generation.
Inside attention, every position produces three projections — a query, a key and a value. The query represents what the current position needs; keys describe what other positions can match on; values carry the information they contribute. The model compares queries with keys, turns the matches into attention strengths, and blends the corresponding values.
Multiple attention heads run in parallel and can capture different relationships — nearby grammar, distant references, patterns in code. It is tempting to imagine each head with one tidy, human-readable job. Neural networks rarely organise themselves that politely.
Each block pairs attention (gathering context) with a feed-forward network (transforming what each position has learned), and residual connections help earlier information survive the trip through many layers. Repeat the block enough times and simple token representations become far richer contextual ones.
Training is not uploading an encyclopedia
During pre-training the model repeatedly plays a next-token prediction game. Give it "The capital of France is" and ask what comes next. Early on, an untrained model is effectively guessing — it might pick "banana".
The system compares predictions against the actual next tokens and calculates a loss: a number representing how wrong it was. Backpropagation works backwards through the computation to determine how each adjustable parameter contributed to the error, and an optimiser nudges those parameters in directions expected to reduce future loss. Then it repeats, on more text. Predict, measure, work backward, adjust.
Those parameters are the weights — billions of them, each just a number. No single weight contains a fact or a skill. Useful patterns are spread across many parameters; knowledge is distributed. The weights are a compressed statistical model of learned patterns, including the data's gaps, contradictions and biases. Models are not searchable copies of their datasets.
Pre-training produces a capable text predictor, not a helpful assistant. Post-training does that: examples of instructions and good responses, then human or model feedback and preference optimisation to make desirable answers more likely. Pre-training learns patterns; post-training teaches the model not to behave like an uncooperative internet autocomplete.
Inference: one token at a time
When you actually use the model, the dials stop moving. The weights stay fixed.
Your prompt becomes tokens, passes through the layers, and the model produces a score — a logit — for every token in its vocabulary. A decoding strategy picks one, appends it, and the whole process runs again. The response is generated token by token.
Everything the model can use for this must sit inside its active context: your prompt, earlier messages, system instructions, tool results. Context is temporary working material — not the trained weights, and not a permanent memory.
Because generation is sequential, reprocessing a long conversation for every new token would be wasteful. Hence one of the least glamorous names in AI: the KV cache. When earlier tokens are processed, their key and value vectors are stored and reused, so only the new query, key and value need computing. It makes generation much faster and consumes substantial memory — an expensive pile of notes, not a diary.
Finally, sampling. Always choosing the highest-scoring token makes output rigid and repetitive, so sampling allows variation. Temperature rescales the logits: higher temperature flattens the distribution and gives less likely candidates more opportunity. It influences variation, not knowledge — turning it down does not make a model better informed.
Why fluent models hallucinate
Now the mechanism explains the failure. Pre-training minimises prediction loss. It does not directly verify whether a claim is true.
If learned patterns support an answer that sounds convincing, the model may generate it even when the claim is wrong. It can blend facts, invent a citation, or describe with total confidence something that never existed. Search, retrieval and tool use can improve grounding — but they attach evidence to a system that was never optimised for truth in the first place.
Fluency is evidence that the model is good at language. It is not evidence that reality approved.
That distinction is the whole basis for treating verification as an operational control rather than a nicety — which is exactly what the EU AI Act's AI-literacy obligation is getting at.
And how it fits on smaller hardware
Weights are stored at a chosen numerical precision. Using fewer bits reduces both the memory they occupy and the data the hardware must move. Imagine recording every dial with extremely fine precision, then deciding fewer positions are accurate enough.
Ignoring overhead: 70 billion weights need roughly 140 GB at 16 bits, but around 35 GB at 4 bits. That is quantisation, and it is a real engineering trade-off — the subject of the local-AI piece.
The whole journey
Text becomes tokens. Tokens become numerical representations. Transformers build context. Training adjusts weights. Post-training shapes the assistant. During inference, frozen weights generate one token at a time, the context window supplies temporary working information, and the KV cache avoids repeating attention work.
None of those pieces is magic on its own. The remarkable behaviour comes from their scale and interaction. What you are working with is a trained prediction system: powerful, imperfect, and entirely capable of saying "certainly" immediately before ruining your afternoon.


