WaferLLM: A Wafer-Scale LLM Inference System

Abstract

Emerging AI accelerators increasingly adopt wafer-scale manufacturing technologies, integrating hundreds of thousands of AI cores in a mesh-based architecture with large distributed on-chip memory (tens of GB) and ultra-high on-chip memory bandwidth (tens of PB/s). However, current LLM inference systems—optimized for shared-memory architectures like GPUs—fail to fully exploit these accelerators. We introduce WaferLLM, the first wafer-scale LLM inference system. WaferLLM is guided by a novel PLMR model (pronounced Plummer), which captures the unique hardware characteristics of wafer-scale architectures. Leveraging this model, WaferLLM pioneers wafer-scale LLM parallelism, optimizing utilization of hundreds of thousands of on-chip cores. It also introduces MeshGEMM and MeshGEMV, the first GEMM and GEMV implementations designed to scale effectively on wafer-scale accelerators. Evaluations show that WaferLLM achieves 200× better wafer-scale accelerator utilization than state-of-the-art systems. On a commodity wafer-scale accelerator, WaferLLM delivers 606× faster and 22× more energy-efficient GEMV compared to an advanced GPU. For LLMs using 16-bit data types, WaferLLM achieves 2700 toks/sec/req on the LLaMA3-8B model and 840 toks/sec/req on the Qwen2-72B model—enabling 39× faster decoding with 1.7× better energy efficiency.