CRAFT: Clustering Rubrics to Diagnose Weak LLM Capabilities and Generate Targeted Fine-Tuning Data
CRAFT is a method that transforms rubric-based evaluation datasets into model-specific diagnoses of weak capabilities by treating each grading criterion as a capability probe. It clusters these capability descriptions into a hierarchical tree, scores the model at each node, and selects low-performing nodes to generate targeted supervised fine-tuning data. In experiments on four open-source models in finance and legal domains, CRAFT outperformed baseline methods in most settings, leading to improved model performance after targeted fine-tuning.
Why it matters: CRAFT enables more precise identification and remediation of model weaknesses, supporting more effective targeted fine-tuning and improved model capabilities.
Full story at: arXiv AI/ML ↗