Skip to main content
§ Video

Local AI, decoded: what 14B, INT8, Q6 and GGUF actually mean

Watch on YouTube ↗

Download a local AI model and you are immediately met with names like Q4_K_M, INT8, 27B, GGUF and TurboQuant. It reads less like artificial intelligence and more like a Wi-Fi password.

Every one of those labels answers a practical question: will this fit on my machine, and what did I give up to make it fit? Here is the vocabulary, in order.


What "local" buys you

Local means the model runs on your own PC or workstation. It lives on your hardware, not in someone else's cloud. You own the compute and you own the data.

The biggest draw is privacy. Your prompts never leave the room and nobody is reading them to improve their next model. For sensitive work — legal, medical, unreleased code, personal data — that is a form of sovereignty the cloud structurally cannot match, and it sidesteps the entire question of whose law governs your provider. It also works offline, and after the hardware there are no per-token bills.

The trade-offs are real. You pay for the hardware up front, and a good GPU is not cheap. You are the administrator: updates, driver conflicts and optimisation are yours. And local models are generally a step behind the largest cloud systems in raw general capability.

One caution worth stating plainly: local does not automatically mean secure. You still need to trust your model sources and your runtime, and both need patching like any other software.


Parameters: billions of dials

Models are measured in B — billions of parameters. Parameters are adjustable numbers learned during training. They are not stored facts or individual neurons; they are mathematical weights. Think of them as billions of tiny dials, each influencing how the model responds.

27B means 27 billion dials. More parameters generally mean more capability — but bigger is not automatically better. How those dials are organised, the architecture, matters as much as the count, and architecture plus training-data quality regularly lets a smaller model punch well above its weight class.

Labels online make this worse, not better. A number in a model's name might refer to total parameters, active parameters, or file size. Always check the model card.

That distinction matters most for mixture-of-experts models. In a dense model, every parameter is used for every token generated — comprehensive, and resource-heavy. An MoE model has a very large total parameter count but activates only a fraction per token: a huge library where only one book is opened at a time. A model can therefore carry a trillion-scale label while using a much smaller number of active parameters per token. The choice between the two has real consequences, and it is not the one most people assume.


Precision and quantisation

Each learned number can be written with more or fewer digits. That is precision: FP16, BF16, INT8 and so on. Lower precision means smaller files and faster arithmetic — but, like a lower-resolution photograph, a less exact model. You trade a little accuracy for a lot of speed and space.

INT8 stores values in eight bits, roughly halving weight memory compared with 16-bit. Runtime memory is not just the weights, though — the context cache and software overhead are on top.

Quantisation is the process of converting those weights to lower precision: packing the suitcase tighter. Higher numbers stay closer to the original. Q8 is close to lossless; Q4 is frequently the sweet spot; Q2 compresses so aggressively that the model starts making mistakes — leaving the shoes behind to save room.

Method matters as much as the number. A high-quality Q4_K_M build is not simply "4-bit, therefore bad". The quantisation scheme is a real part of the quality.

Then there is GGUF: the standard container format for local use. It is the box the compressed model arrives in — robust, widely supported, and not itself the intelligence.


Where the model has to sit

VRAM is the fast workbench next to your GPU. System RAM is the storage room down the hall. When a model does not fit in VRAM, layers get offloaded to system RAM — technically a victory, experientially a disaster, because every word takes seconds to appear.

Modern Macs use unified memory, sharing one pool between CPU and GPU, which simplifies the arithmetic — but you still have to leave room for the operating system.

And every conversation needs a notebook: your context and KV cache, the model's working memory. Longer context means a bigger notebook. This is the trap that catches people out — a model that fits comfortably during a short chat can run out of memory during a long document analysis, because the notebook grew, not the model.

Aggressive KV-cache compression (marketed under names like TurboQuant) can squeeze that notebook down to two or three bits. It saves a great deal of space, but support is hardware-dependent and still experimental. Treat it as an advanced option, not a default.


Choosing by hardware

The video walks a decision ladder from 8 GB up to 128 GB and beyond, with specific model suggestions at each tier — small dense models such as Qwen3.8-27B in the mid-range, and large mixture-of-experts builds like Qwen3.8-2.4T-A95B only where the memory exists to hold the whole model. Those suggestions are explicitly a snapshot — Bert dates them to the day of recording, because local-AI software and model availability move fast enough that any list ages in weeks.

The shape of the ladder is the durable part. At the low end you stay with small models and modest context. In the middle you gain headroom for mid-size models at Q4. At the high end you can run large models at higher precision, and only at the top do the genuinely enormous ones fit at all — where bandwidth, not capacity, becomes the limit.

For a first run: start with a well-supported GGUF model in a Q4_K_M quantisation, set context to a modest 8K, and test the speed before doing anything else. Move up only once you are sure.


Do not chase the biggest number. The best model is the strongest one that fits completely on your workbench, with room left for context and for the machine to keep running.

Model recommendations in the video are a snapshot as of 29 August 2026. Memory use and performance vary by model, quantisation format, context length, runtime and hardware — always check the model card and benchmark on your own machine.