OrderBench: Schema Validity Is Not Enough for Reliable LLM Ordering Agents
A new preprint introduces OrderBench, a benchmark designed to evaluate restaurant ordering agents powered by large language models (LLMs). The study finds that while LLMs can reliably produce outputs that are syntactically and schema-valid, these outputs often contain significant semantic errors. Even the strongest tested model achieved 100% schema validity but only around 80% semantic success, and weaker models exhibited double-digit rates of unsafe acceptances. The authors caution that structured output formats like JSON Schema do not guarantee transaction safety or correctness.
Why it matters: This work exposes a critical limitation in relying solely on schema-constrained outputs for LLM-driven transactional systems, emphasizing the need for additional domain verification to ensure safety and correctness.
Full story at: arXiv AI/ML ↗