The ORCA Benchmark: Evaluating Real-World Calculation Accuracy in Large Language Models

researchers just found that real-world calculation accuracy in large language models is not guaranteed by size or generic math training alone. the orca benchmark is designed to stress real-world tasks where numbers, units, and context matter, not just clean math problems. they found that while some models can handle straightforward arithmetic, performance drops sharply on longer chains or tasks that require maintaining context across steps.

another interesting point is that real-world calculations reveal brittleness in numerical reasoning when external tools or memory are involved; some models rely on internal approximations that break down with precision constraints, leading to surprising errors on seemingly simple tasks. the researchers also note that there’s a big gap between laboratory benchmarks and this real-world oriented evaluation, suggesting that many current models are good at toy problems but stumble in practical calculator-like scenarios. this team provides a benchmark suite that can be used to track progress over time and to highlight where improvements are most needed, such as consistent unit handling, error detection, and robust chaining of calculations.

overall, the paper argues that adding realism to evaluation helps align ai capabilities with practical use cases, and that developers should consider real-world calculation reliability as a key performance axis.

full breakdown: https://www.thepromptindex.com/real-world-calculations-in-ai-how-well-do-todays-language-models-compute-like-a-real-calculator.html

original paper: https://arxiv.org/abs/2511.02589