ScarfBench: Benchmarking AI Agents for Enterprise Java Framework Migration

Jul 10, 2026

IBM Research and Hugging Face have introduced ScarfBench, a benchmark designed to evaluate AI agents on enterprise Java framework migration tasks. ScarfBench features 110 real-world migration tasks from Jakarta EE 8 to Jakarta EE 10, spanning 10 popular open-source projects. Initial results indicate that current AI agents achieve only 10-15% success rates, underscoring the challenges in this domain.

Why it matters: This benchmark provides a standardized way to assess AI agents on complex enterprise software modernization tasks, highlighting current limitations and areas for improvement.

Full story at: Hugging Face Blog ↗