Everything that happens between you hitting enter and the next word appearing — TOPS, precision, VRAM, bandwidth, and why your GPU is built the way it is. Each idea unlocks the next. Play with every widget.
Before we count operations per second, we need to know what one operation is. Strip a neural network down and it's just one move repeated at unimaginable scale: multiply two numbers, add the result to a running total.
TOPS = Tera (trillion) Operations Per Second — how many of those multiply-adds a chip can do. One number, one giant caveat: it's meaningless without the data type it was measured at. 50 TOPS should always be read as 50 TOPS @ INT8.
The headline spec for NPUs and edge AI chips (Apple Neural Engine, Snapdragon, the 40+ TOPS "Copilot+ PC" bar). Almost always quoted at INT8 — halve the bit-width and the number roughly doubles. It's a theoretical peak: real workloads rarely hit it because memory can't feed the units fast enough.
The same idea for floating-point math, the language of training. Quoted in TFLOPS / PFLOPS. Rule of thumb: training talks FLOPS, local inference talks TOPS — and, as you'll see, often talks bandwidth even more.
Lower precision is the lever that makes local AI possible. Fewer bits per weight means a smaller model, more TOPS on the same chip, and a little less quality. This is what GGUF quant levels are.
All those operations have to run on something. The reason a GPU eats LLMs for breakfast comes down to one idea: massive parallelism beats raw single-core speed when the work is millions of independent multiply-adds.
Here's the plot twist that surprises most people: for local chat, your speed usually isn't capped by TOPS at all — it's capped by memory bandwidth. Every token streams the whole model out of VRAM. And VRAM capacity is a separate fight, made harder by the KV cache.
Every concept so far collapses into one practical question. Model size comes from params × precision. Whether it fits is size + KV cache vs VRAM. How fast it runs is bandwidth ÷ size. Turn the dials.
The calculator hands you a tok/stok/s — tokens per second, the user-facing speed metric. Everything in this primer rolls up into this number. number, but a number is abstract. This types out at the real rate, so you can feel the gulf between a model crawling on CPU offload and one living comfortably in VRAM.
You've met all the pieces now — multiply-adds, precision, parallelism, bandwidth. Zoom all the way out and they assemble into a family of chips, each making a different bet on power, price, and who controls the silicon. Same job, very different homes.
Every abbreviation used in this primer, in one place. Hover or tap the dotted terms above for the same definitions inline — nothing here goes undefined.