GPU kernels with Triton

A GPU kernel is a function that runs as many parallel instances on the GPU, and Triton lets you write kernels in Python-like code. The biggest win is usually fusion: doing several steps in one kernel, so that the data leaves slow memory only once.

Why it matters

Many deep-learning operations are memory-bound chains of small steps (normalise, activate, scale). Run as separate PyTorch operations, each one reads and writes the full tensor in slow memory. A fused kernel can be several times faster. Triton makes writing such kernels possible without learning CUDA C++, and it is what torch.compile uses internally.

The idea

A kernel is a small program that runs on the GPU as many parallel copies. Triton lets you write kernels in Python-like code. The biggest win is fusion: doing several steps, like the parts of a softmax, in one kernel, so data is read from slow GPU memory once instead of several times.

Cooking a dish in one trip to the fridge instead of fetching each ingredient separately.

In a kitchen, the fridge trip is the obvious cost. On a GPU you must measure: some operations are limited by the maths, not by memory, and fusion does not help them.

How it works

  1. Split the work into blocks: each kernel instance (a "program") handles one block, for example one row of a matrix.
  2. Each program computes which addresses it owns from its program id, and loads that block with tl.load, using a mask for the ragged edge.
  3. It does all the steps on the block in fast on-chip memory: for softmax, the max, the exponentials, the sum and the division.
  4. It writes only the final result with tl.store.
  5. A naive softmax as separate operations reads the data three times and writes twice; the fused kernel reads once and writes once.
  6. Tune the block size: too small wastes parallel hardware, too large spills out of fast memory.
  7. Always check correctness against PyTorch and benchmark across sizes.

Key points

Common mistakes

Not true: A kernel written by hand is faster than what torch.compile produces.
In fact: torch.compile already fuses many common chains of operations well. Write a kernel by hand only where measurement shows the compiler falls short.

Not true: A kernel needs no mask when the tensor's size is not a multiple of the block size.
In fact: The last block would then read or write past the end of the tensor. A mask switches off the positions that fall outside it.

Part of Unit 6: Systems and scale in the free ML Foundations course.