Topic

Unified KV Cache Management Layer: LMCache’s Open Architecture and Ecosystem Integration(Joint Talk)

As LLM inference evolves from single-engine, single-node deployments toward multi-engine, multi-backend, and cloud-native production environments, KV Cache management is evolving from an internal engine mechanism into an independent system capability. With an open, neutral, and extensible architecture, LMCache provides a unified KV Cache management layer between inference engines and storage backends. This talk will introduce LMCache’s project positioning, open-source community, and core architecture, with a focus on how it enables flexible ecosystem extensions. Using MooncakeStore as an example, the talk will explain how to integrate a new KVStore backend, while SGLang will be used to demonstrate how to integrate an inference engine. The talk will also cover practical considerations for LMCache deployment, observability, debugging, and production operations, followed by a discussion of the future evolution of KV Cache management layers in open inference ecosystems. Outline 1. Introduction to LMCache and Its Open-Source Community Project background, core positioning, applicable scenarios, and LMCache’s role and community status within the open-source inference ecosystem. 2. Architecture of a Unified KV Cache Management Layer How LMCache provides neutral, open, and extensible KV Cache management capabilities between inference engines and storage backends. 3. Extending KVStore: The MooncakeStore Case LMCache’s storage backend extension mechanism, including interfaces, data paths, and key engineering considerations involved in integrating MooncakeStore. 4. Integrating Inference Engines: The SGLang Case How inference engines use LMCache for KV Cache lookup, loading, storage, and reuse, with a focus on scheduling, lifecycle management, and kvgroup adaptation during integration. 5. Deployment, Observability, and Production Operations Practical experience with LMCache configuration, deployment modes, metrics monitoring, log analysis, and troubleshooting in production environments. 6. Summary and Future Outlook Summarize the value of LMCache as a unified KV Cache management layer and discuss its future evolution. Key Takeaways Understand why KV Cache management is evolving from an internal inference engine capability into an independent and unified management layer. Gain a clear understanding of LMCache’s core architecture and how it connects inference engines with storage backends. Learn how to extend LMCache with new KVStore backends, integrate new inference engines, and deploy it in production environments with robust observability and operational support.