PerfReasoning: How Well Do LLMs Reason on Hardware Performance?
Source
arxiv.org
Author
Dan Zhao, Karthikeyan Sankaralingam, Christos Kozyrakis, Qijing Huang
Date
Context
Explaining hardware performance and writing a reliable program to predict it are different skills. PerfReasoning tests both using descriptions of a workload, hardware, and how computation is assigned to it. Models answer questions about data movement and storage, then face the harder task of generating analytical modeling code. The reported results put the strongest closed models above 90% on reasoning questions. For code construction, GPT-5.6 Sol exceeded an 80% pass rate, while every other tested configuration averaged below 15%. Strong verbal reasoning therefore provided an incomplete picture of modeling reliability in this benchmarkA standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.Full definition →.
Terms in this piece · Glossary
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
articleRFCLLM: Evaluating LLMs' Reasoning Ability of Network Protocol State MachinesAnqi Chen, Dan Goldwasser, Cristina Nita-Rotaru
articleDiagnostic Foundation for Evaluating LLMs' Research Integrity as Co-ScientistsYash Tripathi, Silu Sharma, Sai Sidhanth Manoharan Jayanthi, Shivank Garg, Lin Li
articleBeyond the Sycophancy Score: How Task, Model, and Pressure Shape LLM YieldingGuang Yang, Homa Hosseinmardi, Fengchen Liu, Amir Ghasemian