CAFE: A Framework for Evaluating Compound AI Systems Using Design of Experiments

Jul 14, 2026

CAFE is an open-source platform that applies design of experiments to evaluate compound AI systems, enabling practitioners to identify which components most influence answer quality. It uses factorial designs, LLM judges, and mixed-effects models to attribute variance and report effect sizes, significance, and trade-offs. The framework is validated on a retrieval-augmented QA pipeline and is available as a Python package and web app.

Why it matters: CAFE offers a principled and explainable approach for evaluating and optimizing complex AI pipelines with statistical rigor.

Full story at: arXiv Computation and Language ↗