How a GPU works: compute vs memory

A GPU runs thousands of simple cores in parallel. How fast your code runs depends on whether it is limited by the maths (compute-bound) or by moving data to the cores (memory-bound).

Why it matters

Deep learning is only practical because GPUs do enormous amounts of arithmetic in parallel. But many operations barely use that power: they spend their time waiting for data. Knowing which kind of operation you have tells you which optimisation can help — mixed precision, fusion, batching, quantisation — and which will do nothing.

The idea

A GPU has thousands of simple cores that do the same operation on lots of data at once. Often the limit is not the maths but moving data between memory and the cores (memory bandwidth).

A huge kitchen with 1,000 fast chefs but one narrow door for ingredients. The chefs spend a lot of time waiting at the door.

Chefs can each cook a different dish. GPU cores work best when they all do the same operation on different data.

How it works

  1. A GPU has thousands of cores grouped into streaming multiprocessors. They all run the same instruction on different data at once.
  2. Data lives in large GPU memory (HBM), which is fast by normal standards but slow compared with the cores. Each multiprocessor also has a small, very fast on-chip memory (registers and shared memory, or SRAM).
  3. Every operation must move its inputs from HBM to the cores and its outputs back.
  4. Arithmetic intensity = FLOPs done ÷ bytes moved. Adding two vectors does 1 addition for every 2 numbers loaded and 1 stored: very low intensity, so it is memory-bound.
  5. Multiplying two big matrices reuses every loaded number many times: high intensity, so it is compute-bound — what GPUs are built for.
  6. The roofline picture: below a certain intensity, speed is capped by memory bandwidth; above it, by peak compute.
  7. So: memory-bound work gets faster by moving fewer bytes (fusing operations, using 16-bit or 8-bit numbers); compute-bound work by using tensor cores and lower precision maths.

Key points

Common mistakes

Not true: Your code runs at the GPU's advertised peak number of operations per second.
In fact: The peak is reached only while the chip is kept busy with maths. Most real code spends its time moving data in and out of memory, so it runs far below the peak.

Not true: A memory-bound operation gets faster when the GPU has more compute.
In fact: A memory-bound operation is limited by how fast data moves in and out of memory. Extra compute sits idle: only moving less data makes it faster.

Part of Unit 6: Systems and scale in the free ML Foundations course.