.png)
Hugging Face published “Profiling in PyTorch (Part 3): Attention is all you profile”, the third entry in its profiling series. It uses attention as the next step after basic ops and MLPs, and shows how different attention implementations surface in PyTorch profiler traces.
The post starts with the textbook causal attention path: q @ k.T, scale, mask with masked_fill, softmax, then softmax @ v. That version is useful mostly as a baseline, because the profiler makes its inefficiency obvious.
The important part is not that attention is expensive in the abstract; it’s that the trace shows where the time goes at the operator level. In a naive implementation, you end up with a sequence of separate kernels and intermediate tensors, which is exactly the kind of pattern profiling is meant to expose. For ML engineers reading traces in production, this is the sort of baseline that tells you whether a performance issue is caused by algorithmic structure, kernel launch overhead, or just a bad implementation choice.
The next step is a version that uses in-place operations where possible. This is not a magic optimization, but it is the first example in the series where the trace becomes a practical guide to reducing overhead without changing model behavior.
That distinction matters. If a profiler trace shows a lot of small tensor ops surrounding the actual compute-heavy steps, you should expect limited gains from “faster hardware” alone. The win comes from collapsing work, reducing temporary allocations, and cutting down dispatch overhead. In other words, the trace tells you whether the implementation is leaving easy performance on the table.
The interesting part of the post is the move to scaled_dot_product_attention (SDPA). This is where the blog stops being a “how to read traces” exercise and becomes a map of what modern attention optimization actually looks like in PyTorch.
SDPA changes the profiler picture dramatically because it replaces the hand-assembled sequence of ops with a higher-level primitive that can route to better implementations. That’s the right mental model for production work: if the profiler shows a forest of separate matmuls, masks, and softmax calls, you are probably not using the most efficient attention path available in your stack.
What I’d pay attention to here is not just raw latency, but the shape of the trace. SDPA is valuable because it reduces visible fragmentation in the execution timeline. When you are debugging throughput regressions or GPU underutilization, a cleaner trace is often the first sign that the model is executing a more optimized code path.
The last step in the post is the kernel-level view. That’s the part that matters most if you’re shipping models at scale, because once you’ve moved past the naive formulation, the performance story is mostly about which kernel gets selected under which conditions.
This is where I’d be skeptical of any attention benchmark that reports a single number without showing the trace. Attention performance is extremely sensitive to implementation details, shape regime, and backend selection. A “fast” result that comes from one kernel in one configuration does not tell you much about behavior across sequence lengths, head dimensions, or deployment settings.
For engineers evaluating attention performance, the right workflow is:
That sequence is more useful than chasing a headline speedup, because it tells you whether you are improving the algorithmic path or just optimizing one narrow case.
The series continues to do the most valuable thing profiling tutorials can do: it trains you to read traces as a causal story, not just as a performance report. For attention specifically, that skill is the difference between guessing at optimizations and knowing whether your model is actually running on the fast path.
