Research→Official→arXiv Software Engineering
UniCode is a generative evaluation framework designed to systematically probe the code reasoning abilities of large language models (LLMs). It introduces multi-dimensional augmentation of coding problems, automated test generation, and fine-grained metrics to reveal model limitations. Experiments show that state-of-the-art models experience a 31.2% performance drop on UniCode, mainly due to weaknesses in conceptual modeling and scalability reasoning.
Why it matters: UniCode demonstrates that current coding benchmarks may overstate LLM capabilities by allowing reliance on statistical shortcuts rather than genuine reasoning.
Jul 16, 2026
Research→Official→arXiv Software Engineering
A longitudinal case study compares API-based Claude Opus with on-premise GLM-5.1/5.2 for enterprise coding agents over two 28-day periods. Prompt caching achieved a 99.3% hit rate, reducing API costs by 88.6% to $0.57 per million tokens, which is lower than the amortized unit cost of the on-premise solution. While on-premise deployment saved 40.1% of total cost of ownership (TCO) under shared GPU allocation, it resulted in a higher defect-repair burden, with a Fix Commit Ratio of 74.9% versus 45.9% for the API-based approach.
Why it matters: This study provides empirical data on the cost and quality trade-offs between cloud and on-premise LLM deployments for enterprise coding agents, offering practical insights for infrastructure decision-making.
Jul 16, 2026
Research→Official→arXiv Software Engineering
SemaDiff is a novel approach that uses large language model (LLM)-generated code and tests to distinguish between semantic-preserving and behavior-changing commits in software repositories. By generating additional dependent classes and tests for both pre- and post-commit code versions, SemaDiff can detect behavioral differences. In evaluation on a manually annotated dataset of 183 Java commits, SemaDiff achieved 76% accuracy and 100% precision in identifying semantic-changing commits.
Why it matters: This method advances software repository mining by enabling more reliable identification of purely refactoring commits, which is important for tasks such as debugging, fault localization, and constructing bug datasets.
Jul 16, 2026
Research→Official→arXiv Software Engineering
A new method called Design-Specification Tiling (DST) is proposed for selecting in-context learning exemplars in CAD code generation tasks. DST frames exemplar selection as a submodular maximization problem, providing a (1-1/e)-approximation guarantee, and aims to maximize coverage of design requirements. Experiments across multiple large language models show that DST leads to substantial improvements in CAD code generation quality compared to existing selection strategies.
Why it matters: This work introduces a principled approach to exemplar selection that enhances the effectiveness of LLMs in complex, domain-specific code generation tasks like CAD.
Jul 16, 2026
Research→Official→arXiv Software Engineering
VisualRepair is a new framework for automated program repair that leverages multimodal large language models (MLLMs) to incorporate visual information from bug screenshots. It introduces image type-aware tool calling and dynamic region focusing to improve fault localization and patch generation. On the SWE-bench Multimodal benchmark, VisualRepair resolves 196 test set instances, outperforming the best baseline by 10 instances.
Why it matters: This work demonstrates a meaningful advance in automated program repair by effectively integrating visual information from bug reports, addressing challenges in modern software with graphical interfaces.
Jul 16, 2026
Research→Official→arXiv Software Engineering
Researchers have introduced GameEngineBench, a benchmark designed to evaluate coding agents on C++ implementation tasks within Unreal Engine 5 projects. The benchmark comprises 110 tasks sourced from nine real-world game repositories, covering a wide range of gameplay and engine-related challenges. The best-performing model achieved only 55.5% pass@1, and 31 tasks were not solved by any tested configuration, underscoring the difficulty of these tasks for current AI coding agents.
Why it matters: This benchmark highlights a significant gap in the ability of current coding agents to handle complex, integrated C++ development in real-time interactive environments, pointing to important limitations in AI-assisted software engineering.
Jul 16, 2026
Research→Official→arXiv Software Engineering
Researchers introduce Monty, a framework that leverages large language models (LLMs) to automatically formalize natural-language assertions into executable formal contracts. Monty improves precision by up to 20 points over naive LLM translation by filtering candidate formalizations using conformance and validity scores. The system was evaluated on 541 assertion-generation tasks from 22 Java collection-like classes.
Why it matters: Automating the creation of formal contracts could reduce the manual effort required for software testing and verification, potentially enhancing developer productivity and software reliability.
Jul 16, 2026
Research→Official→arXiv Software Engineering
VulWeaver is a novel approach that combines large language models (LLMs) with structured program analysis to improve vulnerability detection in source code. It achieves F1-scores of 0.76 on Java and 0.78 on C/C++ datasets, outperforming existing baselines by 17-25%. In real-world tests, VulWeaver detected 26 true vulnerabilities across 9 Java projects, with 15 confirmed by developers and 5 CVEs assigned.
Why it matters: VulWeaver demonstrates that integrating LLMs with program analysis can significantly advance automated vulnerability detection, with practical impact validated by developer confirmations and CVE assignments.
Jul 16, 2026
Research→Official→arXiv Software Engineering
Researchers introduce PROBE, a benchmark framework for evaluating code generation by large language models (LLMs) across multiple dimensions: functional correctness, proximity to valid solutions, and code quality. PROBE assesses models using five programming languages and three prompting strategies, and includes analysis of common errors. The results show that LLMs struggle with more difficult problems and that smaller models perform worse on low-resource languages, often making fundamental errors.
Why it matters: PROBE enables a more thorough and systematic evaluation of LLM-generated code, revealing persistent reliability challenges that are not captured by existing benchmarks.
Jul 16, 2026
Research→Official→arXiv Software Engineering
SeedSmith is an agentic LLM pipeline designed to generate initial seed inputs for directed fuzzing by emulating a security analyst's workflow. It iteratively explores codebases, resolves indirect calls, identifies crash preconditions, and synthesizes concrete inputs, serving as a fuzzer-agnostic seed generation front-end. In evaluations, SeedSmith enabled fuzzers to achieve crash-time speedups of 11.51x to 14.66x on Magma and to trigger 16 previously unreachable bugs across 10 projects with diverse input formats.
Why it matters: SeedSmith demonstrates a significant advance in automating and improving the efficiency of vulnerability discovery through LLM-driven seed generation for fuzzing.
Jul 16, 2026
Research→Official→arXiv Software Engineering
A controlled experiment at a large Brazilian enterprise found that design-system-aware AI assistance reduced front-end development time by 46.7% to 69.4% across Angular, iOS, and Android stacks. The study also observed increased task completeness and reduced performance variability compared to manual or design-system-only development. Analysis indicated that AI tools helped reduce workflow friction and improved the consistency of design implementation.
Why it matters: This study provides empirical evidence that integrating AI with design systems can substantially improve the speed, consistency, and quality of industrial front-end development workflows.
Jul 16, 2026
Policy Safety→Official→arXiv Software Engineering
Researchers introduce FairCoder, a benchmark that frames high-stakes decision-making as coding tasks to systematically probe large language model (LLM) bias in domains such as employment, education, and healthcare. The study also proposes FairScore, a new metric that accounts for both refusal behavior and group-level outcome diversity. Experiments on leading LLMs reveal consistent and previously underexplored bias patterns, including a tendency to prioritize applicants from high-income families in college admissions. The work provides a comprehensive framework for evaluating LLM bias in practical decision-making scenarios.
Why it matters: This research highlights the risks of deploying LLMs in real-world decision-making and offers tools for more thorough bias evaluation.
Jul 16, 2026
Research→Official→arXiv Software Engineering
A new framework enables LLM-based coding agents to persistently learn from human review feedback by codifying accepted review comments as behavioral rules, without requiring model retraining. Deployed across a 35+ service microservices platform, the rule set expanded from 5 to 18 behavioral rules, eliminating recurrence of previously ruled-against error classes. The system shifts human review focus from low-level correctness to higher-level design validation.
Why it matters: This approach demonstrates a practical method for coding agents to continuously improve and reduce repetitive errors in real-world codebases without retraining.
Jul 16, 2026
Policy Safety→Official→arXiv Software Engineering
A new preprint proposes 'falsifiable release gates' for self-improving AI systems, requiring each new capability to pass a machine-verifiable acceptance suite before deployment. The methodology is demonstrated on the Antahkarana open runtime, using seven gates to ensure safety invariants are maintained. The approach is open-sourced for reproducibility and is designed to be adaptable to other agent frameworks.
Why it matters: This work introduces a concrete, reproducible method for verifying safety in self-improving AI systems, addressing a key challenge in AI safety engineering.
Jul 16, 2026
Research→Official→arXiv Software Engineering
Researchers have introduced XVada, a method for black-box context-free grammar inference that outperforms existing approaches like TreeVada in both accuracy and compactness. In empirical evaluations, XVada discovered a CVE in the Python Liquid engine and identified five additional bugs through fuzzing, all of which were subsequently fixed by the developers.
Why it matters: This work demonstrates a significant advance in automated grammar inference with practical impact, as evidenced by the discovery and remediation of real-world software vulnerabilities.
Jul 15, 2026
Research→Official→arXiv Software Engineering
A study of 22 Triton and TileLang GPU kernels reveals that correctness-based evaluation can overlook severe performance issues—one TileLang LayerNorm kernel, for example, passes correctness checks but is over 300× slower than the PyTorch baseline. The underlying causes of inefficiency vary by kernel family, with some due to repairable authoring defects and others stemming from code-generation or autotuning limitations. The authors propose two lightweight screening methods—library-relative efficiency and roofline utilization—that successfully identify all functionally correct but inefficient kernels in their evaluation.
Why it matters: This work exposes a critical flaw in current GPU kernel evaluation practices and offers practical tools to ensure that functionally correct kernels also meet performance expectations.
Jul 15, 2026
Research→Official→arXiv Software Engineering
Researchers have introduced PerfCodeBench, a benchmark designed to evaluate large language models (LLMs) on system-level high-performance code optimization tasks. PerfCodeBench features executable correctness checks, baseline implementations, and reference optimized solutions, enabling assessment of both code correctness and runtime efficiency. Results show a significant gap between model-generated code and expert-optimized solutions, particularly in tasks involving parallelism and GPU operations.
Why it matters: PerfCodeBench exposes the limitations of current LLMs in generating efficient, hardware-aware code for performance-critical systems, emphasizing the need for performance-oriented evaluation in code generation.
Jul 15, 2026
Research→Official→arXiv Software Engineering
RESOURCE2SKILL is a framework that extracts and organizes executable skills for software agents from multimodal resources such as tutorial videos, code repositories, and articles. These skills are structured in a hierarchical Skill Wiki that integrates text, code, visual examples, and metadata, enabling agents to retrieve and compose relevant skills for complex tasks. In evaluations across seven authoring domains, RESOURCE2SKILL improved agent performance by an average of 11.9 percentage points compared to agents without skill libraries, and outperformed strong baselines in most tested scenarios.
Why it matters: This work demonstrates a significant advance in enabling software agents to autonomously acquire and utilize diverse, human-created resources as reusable skills, enhancing their adaptability and effectiveness in complex tasks.
Jul 15, 2026
Research→Official→arXiv Software Engineering
Researchers introduce ArtifactGuide, a structured rubric for software artifact evaluation, and ArtifactCopilot, an LLM-based agent that automates the evaluation process. In tests on 60 real software artifacts, ArtifactCopilot achieved 70.56% badge-level agreement with human reviewers, outperforming baseline prompt-based approaches. A user study with 8 experienced researchers found that ArtifactCopilot's reports improved reviewer confidence and helped locate supporting evidence.
Why it matters: This work demonstrates a significant advance in automating software artifact evaluation, potentially reducing reviewer workload and improving consistency in the review process.
Jul 15, 2026
Policy Safety→Official→arXiv Software Engineering
A new preprint introduces a vendor-neutral metric designed to evaluate whether evidence from agent-safety evaluations can reconstruct the decisions underlying safety claims. The metric assesses reconstructability across eight decision-property classes and includes a cross-harness adapter for generating Evidence Sufficiency Cards. Tests on public traces show sufficiency scores between 0.458 and 0.833, with replay preconditions unmet in all scored traces.
Why it matters: This work provides a standardized approach to evaluating the validity of agent safety claims, addressing a key challenge in building trustworthy AI systems.
Jul 15, 2026