GPU kernels with Triton
A GPU kernel is a function that runs as many parallel instances on the GPU, and Triton lets you write kernels in Python-like code. The biggest win is usually fusion: doing several steps in one kernel, so that the data leaves slow memory only once.
Why it matters
Many deep-learning operations are memory-bound chains of small steps (normalise, activate, scale). Run as separate PyTorch operations, each one reads and writes the full tensor in slow memory. A fused kernel can be several times faster. Triton makes writing such kernels possible without learning CUDA C++, and it is what torch.compile uses internally.
The idea
A kernel is a small program that runs on the GPU as many parallel copies. Triton lets you write kernels in Python-like code. The biggest win is fusion: doing several steps, like the parts of a softmax, in one kernel, so data is read from slow GPU memory once instead of several times.
Cooking a dish in one trip to the fridge instead of fetching each ingredient separately.
In a kitchen, the fridge trip is the obvious cost. On a GPU you must measure: some operations are limited by the maths, not by memory, and fusion does not help them.
How it works
- Split the work into blocks: each kernel instance (a "program") handles one block, for example one row of a matrix.
- Each program computes which addresses it owns from its program id, and loads that block with tl.load, using a mask for the ragged edge.
- It does all the steps on the block in fast on-chip memory: for softmax, the max, the exponentials, the sum and the division.
- It writes only the final result with tl.store.
- A naive softmax as separate operations reads the data three times and writes twice; the fused kernel reads once and writes once.
- Tune the block size: too small wastes parallel hardware, too large spills out of fast memory.
- Always check correctness against PyTorch and benchmark across sizes.
Key points
- Kernel = one program, run as many parallel instances over blocks of data.
- Fusion cuts memory traffic for memory-bound chains.
- Masks handle sizes that aren't a multiple of the block size.
- Block size is a key tuning knob.
- torch.compile can often fuse operations for you automatically.
- Compute-bound operations (big matmuls) gain little from fusion.
Common mistakes
- Forgetting the mask, so the last block reads or writes past the end of the tensor.
- Benchmarking without warm-up runs and torch.cuda.synchronize(), so the timing includes start-up costs or stops before the GPU has finished.
- Hand-writing a kernel for something torch.compile already fuses well.
- Not checking numerical stability (for softmax, subtract the max inside the kernel too).
Not true: A kernel written by hand is faster than what torch.compile produces.
In fact: torch.compile already fuses many common chains of operations well. Write a kernel by hand only where measurement shows the compiler falls short.
Not true: A kernel needs no mask when the tensor's size is not a multiple of the block size.
In fact: The last block would then read or write past the end of the tensor. A mask switches off the positions that fall outside it.
Part of Unit 6: Systems and scale in the free ML Foundations course.