Topic

System Optimization and Production Practices for Long-Context Inference with Xiaomi MiMo

Long-context capabilities expand the possibilities of multi-turn interactions, coding agents, and multimodal understanding, but they also introduce challenges for inference services in computational overhead, KV cache capacity, data transfer, and request scheduling. In real-world production environments, requests with similar context lengths can still differ significantly in cache hit rates, computational requirements, and resource utilization. Improving throughput and reducing costs while meeting latency constraints has therefore become a key system optimization challenge. Drawing on Xiaomi MiMo’s long-context inference practices based on SGLang, this talk explores KV cache management under hybrid attention, multi-level cache reuse and transfer optimization, dynamic bucketing based on context length and cache hit rates, and scheduling strategies that balance computational load with cache affinity. It also examines how these mechanisms interact with Prefill–Decode disaggregation, speculative decoding, and low-precision execution. In addition, the talk discusses end-to-end resource allocation and performance trade-offs in multimodal encoder disaggregated deployment. Through representative engineering case studies, the talk presents the practical process from bottleneck identification and solution validation to gradual production rollout. It demonstrates how to evaluate the system-level benefits of individual optimizations under real-world workloads, offering practical insights into improving the performance, cost efficiency, and reliability of long-context inference services. Talk Outline: 1. Workload Characteristics and System Bottlenecks in Long-Context Inference: Computation, GPU Memory, Data Transfer, and Queueing Latency. 2. Cache and Memory Management: Hybrid Attention, Multi-Level KV Cache, Cache Reuse, and Cross-Node Data Transfer. 3. Bucketing and Scheduling Optimization: Dynamic Bucketing, Isolation of Low-Cache-Hit Requests, and Trade-Offs Between Load Balancing and Cache Affinity. 4. End-to-End Optimization and Production Validation: Disaggregated Deployment, Speculative Decoding, Low-Precision Execution, Performance and Accuracy Validation, and Gradual Rollout Practices. Key Takeaways: - Understand how computation, caching, and scheduling interact in long-context inference systems. - Learn approaches to designing bucketing and scheduling strategies based on request characteristics. - Understand how to evaluate whether localized performance improvements translate into measurable throughput gains and cost savings in production environments.