Profiling PyTorch code
Profiling measures where time is actually spent in your code — CPU, GPU, data loading, copies — so you optimise the real bottleneck instead of guessing.
Why it matters
Intuition about performance is often wrong: the slow part of training may be data loading, a tiny operation called thousands of times, or a hidden copy between CPU and GPU. Optimising the wrong thing wastes days. A profiler shows the timeline and tells you exactly where to look.
The idea
Measure where the time actually goes before you optimise. The profiler shows the time for each operation on the CPU and GPU. Common findings: the GPU waiting for data loading, many tiny operations, or slow copies between CPU and GPU.
Timing each station in a kitchen before hiring more chefs. Maybe the slow part is the dishwasher.
Kitchen timing is simple. GPU work runs asynchronously, so normal Python timers can lie. Call torch.cuda.synchronize() before you read the time.
How it works
- Wrap a few training steps in torch.profiler.profile(activities=[CPU, CUDA]), after a couple of warm-up steps.
- Look at the table of operations sorted by total time on the CPU and on the GPU.
- Open the timeline trace (in a browser trace viewer or TensorBoard) to see, over time, what the CPU and GPU are each doing.
- Gaps on the GPU row mean it is waiting — usually for data loading or Python overhead.
- Many tiny kernels mean launch overhead: fuse operations or use torch.compile.
- Large CPU-to-GPU copies inside the loop mean data is being moved too often; keep tensors on the GPU and use pinned memory.
- Fix one thing, profile again, and compare steps per second.
Key points
- Measure first, optimise second: the slow part is often not the one you would have guessed.
- GPU work is asynchronous: call torch.cuda.synchronize() before reading a timer.
- Warm up before measuring; the first steps include one-off setup costs.
- Common bottlenecks: data loading, tiny operations, CPU–GPU copies, synchronisation points like .item().
- More DataLoader workers and pin_memory=True often remove data-loading stalls.
Common mistakes
- Timing GPU code with time.time() without synchronising.
- Profiling the very first steps, which include compilation and caching.
- Calling .item() or printing every step, which forces the CPU to wait for the GPU.
- Optimising an operation that takes 2% of the time.
Not true: Optimising an operation that takes 2% of the time: a noticeable speed-up for the whole program.
In fact: An operation that takes 2% of the time can save at most 2%. Find the biggest share of the time first, and optimise that.
Not true: Timing GPU code: time.time() before and after the call gives how long the GPU took.
In fact: GPU work is asynchronous: the call returns before the work is finished. Call torch.cuda.synchronize() before reading the timer.
Part of Unit 6: Systems and scale in the free ML Foundations course.