QuArch: A Benchmark for Evaluating LLM Reasoning in Computer Architecture
Researchers have introduced QuArch, the first benchmark specifically designed to evaluate large language models' (LLMs) knowledge and reasoning abilities in computer architecture. QuArch includes 2,671 expert-validated question-answer pairs spanning topics such as processor design and memory systems. Leading LLMs achieved between 34% and 73% accuracy on advanced questions, and fine-tuning on QuArch led to improved performance on a realistic memory hierarchy design task, yielding more area-efficient and viable hardware solutions.
Why it matters: QuArch addresses a significant gap in LLM evaluation by providing a domain-specific benchmark for computer architecture, enabling more rigorous assessment and development of AI capabilities in this critical field.
Full story at: arXiv Software Engineering ↗