Dr Ashish Bamania · LLM Inference Internals

A Hardware-Level Tour of LLM Inference

Traces inference down to the metal: weights moving from SSD through CPU DRAM into GPU HBM over PCIe, the CPU-HBM-SRAM hierarchy, and the tensor, pipeline, context, expert and data parallelism used to spread work across GPUs. It separates the compute-bound prefill phase, where the whole prompt is processed at once, from the memory-bound decode phase producing one token at a time, and explains the KV cache as what prevents the recomputation. The argument it builds to is that memory bandwidth, not raw compute, is the wall.