Abstract:
In large language model (LLM) serving, each request accumulates persistent memory during service as its key–value cache grows with every generated token. Under high concurrency, aggregate memory usage therefore increases endogenously over time: service itself creates future capacity pressure. This progressive resource consumption breaks the monotonicity assumptions underlying classical stability analyses and gives rise to a new form of service-induced congestion. In this work, we develop a discrete-time dynamical model of memory-constrained inference that captures the interplay among admission, memory growth, and throughput. In a saturated-input regime, the system admits both eviction-free fixed points and periodic limit cycles. For homogeneous job sizes, we show that the eviction-free equilibrium is unstable under standard continuous batching and that the dynamics converge to a unique asymptotically stable worst-case limit cycle, where throughput losses can be as large as 50%. For heterogeneous job sizes, we establish a sharp stability dichotomy: under a large-prompt scaling, the eviction-free equilibrium is asymptotically stable if and only if the decoding lengths are coprime. Our results characterize when workload heterogeneity stabilizes the system and when it fails to do so. Together, these findings identify service-induced congestion as a structural instability mechanism in memory-constrained systems and provide conditions under which heterogeneity restores stability.
About the Speaker:
Jing Dong is the DeRosa Family Associate Professor of Business in the Decision, Risk, and Operations Division at Columbia Business School and a visiting faculty member at the Antai College of Economics and Management at Shanghai Jiao Tong University. Her research lies at the intersection of applied probability and service operations management, with a particular focus on patient flow management in healthcare delivery systems. She serves as an associate editor for Operations Research, Mathematics of Operations Research, Management Science, Manufacturing & Service Operations Management, and the INFORMS Journal on Computing, and as vice chair of the INFORMS Applied Probability Society. She received her Ph.D. in Operations Research from Columbia University and was on the faculty at Northwestern University before joining Columbia Business School.
Your participation is warmly welcomed!
欢迎扫码关注北大统计科学中心公众号，了解更多讲座信息!
