Alipay-PIBench: A Realistic Payment Integration Benchmark for Coding Agents

Jul 17, 2026

Alipay-PIBench is a new benchmark designed to evaluate coding agents on realistic Alipay payment integration tasks. It features nine product-specific projects and 18 task instances, covering both basic functional-completion and advanced risk-aware hardening scenarios. The benchmark uses rubric-based and LLM-assisted assessments to evaluate agent performance. Experiments with six coding-agent models show mean rubric pass rates from 68.58% to 91.37% under the with-skill condition, and access to the payment integration skill improves performance by an average of 10.31 percentage points.

Why it matters: This benchmark enables controlled, realistic evaluation of coding agents on complex, multi-step payment integration tasks, which is important for advancing AI deployment in financial software development.

Full story at: arXiv AI/ML ↗