How a GPU works: compute vs memory
A GPU runs thousands of simple cores in parallel. How fast your code runs depends on whether it is limited by the maths (compute-bound) or by moving data to the cores (memory-bound).
Why it matters
Deep learning is only practical because GPUs do enormous amounts of arithmetic in parallel. But many operations barely use that power: they spend their time waiting for data. Knowing which kind of operation you have tells you which optimisation can help — mixed precision, fusion, batching, quantisation — and which will do nothing.
The idea
A GPU has thousands of simple cores that do the same operation on lots of data at once. Often the limit is not the maths but moving data between memory and the cores (memory bandwidth).
A huge kitchen with 1,000 fast chefs but one narrow door for ingredients. The chefs spend a lot of time waiting at the door.
Chefs can each cook a different dish. GPU cores work best when they all do the same operation on different data.
How it works
- A GPU has thousands of cores grouped into streaming multiprocessors. They all run the same instruction on different data at once.
- Data lives in large GPU memory (HBM), which is fast by normal standards but slow compared with the cores. Each multiprocessor also has a small, very fast on-chip memory (registers and shared memory, or SRAM).
- Every operation must move its inputs from HBM to the cores and its outputs back.
- Arithmetic intensity = FLOPs done ÷ bytes moved. Adding two vectors does 1 addition for every 2 numbers loaded and 1 stored: very low intensity, so it is memory-bound.
- Multiplying two big matrices reuses every loaded number many times: high intensity, so it is compute-bound — what GPUs are built for.
- The roofline picture: below a certain intensity, speed is capped by memory bandwidth; above it, by peak compute.
- So: memory-bound work gets faster by moving fewer bytes (fusing operations, using 16-bit or 8-bit numbers); compute-bound work by using tensor cores and lower precision maths.
Key points
- GPUs are throughput machines: huge parallelism, same operation on lots of data.
- Memory bandwidth is often the real limit, not the FLOPs.
- Arithmetic intensity decides memory-bound versus compute-bound.
- Matrix multiplies are compute-bound; element-wise operations, norms and softmax are usually memory-bound.
- Many tiny operations are slow because of launch overhead and memory traffic; fewer, bigger operations are better.
- Tensor cores make 16-bit and 8-bit matrix maths much faster than 32-bit.
Common mistakes
- Assuming the GPU's peak FLOPs is what your code achieves.
- Trying to speed up a memory-bound operation with more compute.
- Launching thousands of tiny kernels in a Python loop.
- Copying data between CPU and GPU inside the training loop.
Not true: Your code runs at the GPU's advertised peak number of operations per second.
In fact: The peak is reached only while the chip is kept busy with maths. Most real code spends its time moving data in and out of memory, so it runs far below the peak.
Not true: A memory-bound operation gets faster when the GPU has more compute.
In fact: A memory-bound operation is limited by how fast data moves in and out of memory. Extra compute sits idle: only moving less data makes it faster.
Part of Unit 6: Systems and scale in the free ML Foundations course.