ContinuityBench: Benchmark and Systems Study of Stateful Failover in Multi-Provider LLM Routing
Researchers introduce ContinuityBench, a benchmark and systems study focused on maintaining conversational continuity during failover between large language model (LLM) providers. They propose a stateful proxy architecture that uses a History-Forwarding strategy to reconstruct conversational state across heterogeneous LLM endpoints, achieving a 99.20% Continuity Preservation Rate (CPR) compared to near-0% for stateless systems. The work also includes an open-source evaluation harness and new metrics for robust multi-provider LLM routing.
Why it matters: This work establishes a principled approach for building robust, state-preserving multi-model inference systems, which is critical for reliable production LLM deployments.
Full story at: arXiv Machine Learning ↗