Model Instability in Software Analytics Can Be Quantified and Reduced, Study Finds

Jul 14, 2026

A new preprint reports that model instability is a significant issue in software analytics, with repeated runs of the same optimizer agreeing on only 13.7% of test cases across 127 multi-objective software engineering optimization problems. The researchers show that by adjusting label allocation, model complexity, and scoring splits, agreement can be improved by a factor of 4.8 and the standard deviation of optimization error reduced by 22% on average, without degrading recommendation quality. The study argues that instability is not just noise but a measurable and manageable property, and recommends treating it as a standard evaluation axis.

Why it matters: This work provides a practical framework for measuring and managing model instability, which could enhance the reliability and trustworthiness of software engineering optimization tools.

Full story at: arXiv Software Engineering ↗