.png)
AWS announced disaggregated prefill and decode (DPD) for LLM inference on Amazon SageMaker HyperPod, implemented with vLLM and the HyperPod Inference Operator. The release splits prompt processing and token generation onto separate GPU pools connected by EFA with RDMA, so long prefills stop blocking decode traffic.
The core issue is resource mismatch. Prefill is compute-bound and front-loads the prompt into KV cache. Decode is memory-bandwidth-bound and runs token by token against model weights plus an ever-growing cache. When both phases share the same GPUs, a single long prompt can inflate TTFT and spike ITL for unrelated in-flight requests.
DPD removes that interference by turning prefill and decode into separate inference engines. That gives each phase its own parallelization strategy, so TTFT and ITL can be tuned independently instead of forcing one compromise setting for the whole request path. For workloads with mixed prompt lengths, that matters more than another incremental batching tweak.
AWS is targeting the class of workloads where the pain is obvious: chat assistants, agentic systems, document analysis, and RAG endpoints with large retrieved contexts. In those systems, prompt lengths are uneven and concurrency is high enough that colocated prefill becomes a head-of-line blocker.
vLLM already handles single-node efficiency well through continuous batching and PagedAttention. The new piece is the multi-node serving architecture needed when prefill and decode run on separate pools. The AWS setup uses the HyperPod Inference Operator to manage the deployment and lifecycle of those components on a SageMaker HyperPod cluster.
The important systems detail is the network path between the two pools. EFA with RDMA is doing the heavy lifting for moving data between stages without turning the handoff into the new bottleneck. That matters because disaggregation only helps if the transport layer is fast enough to preserve the gains from specialization.
This is also where DPD differs from the usual chunked-prefill tuning game. Chunking can reduce blocking, but it still couples the phases on the same execution resources. DPD enforces isolation by construction, so the long-context request does not contend with decode on the same GPU.
This is not the default choice for every serving stack. DPD makes the most sense when prompt lengths are often large, concurrency is meaningful, and output latency is user-visible. If requests are short and homogeneous, the extra distributed-system complexity may buy little.
I would evaluate DPD first when: - prompts regularly exceed 4,096 tokens, - request concurrency is high enough that decode stalls are measurable, - TTFT and ITL are both SLOs, not just throughput, - long-context requests arrive alongside short interactive traffic.
That profile is common in production RAG and tool-using assistants. It is also common in enterprise document workflows, where retrieved context can dominate prompt length and create pathological interference for everyone else sharing the same serving pool.
What I would want to validate before adopting it broadly is whether the EFA/RDMA path and the operational overhead of maintaining two GPU pools are justified by the tail-latency reduction on my real traffic mix. For mixed workloads with ugly prompt distributions, the answer is often yes. For simpler workloads, the extra topology is probably overkill.
