PHITSBench: Benchmark for AI-Assisted Radiation Transport Code Generation

Jul 14, 2026

Researchers introduced PHITSBench, a benchmark for evaluating AI-assisted generation of PHITS radiation transport code from natural language. GPT-5.4 achieved 95% success on parameter editing and 70% on syntax repair tasks, but 0% on generating simulations from scratch without domain knowledge. Providing a structured knowledge catalog increased single-shot success on simulation generation to 57%, and agentic workflows further improved it to 66-73%.

Why it matters: This benchmark demonstrates that current AI models struggle to generate correct scientific simulations from natural language without machine-readable knowledge, highlighting the importance of structured knowledge bases and execution-grounded evaluation.

Full story at: arXiv AI/ML ↗