CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models

A model can reproduce the direction of a human effect while making different errors. CogBattery evaluates 54 open-weight language models on text adaptations of 13 cognitive paradigms and compares each with human data from its source study. Models respond more to n-back lures than to controls, as humans do, yet make far more false alarms on ordinary non-targets. Most separate correct from incorrect answers less well than humans do in their reported confidence, and most always choose the same wager. Across tasks, model scores share two dimensions; the first is closely tied to scale, yet larger models are not consistently closer to humans, and predefined cognitive groupings show no reliable predictive advantage over random groupings of the same size.

Code: github.com/dengzhe-hou/CogBattery. First released as CogArena (arXiv v1) and renamed in v2 to avoid confusion with an unrelated benchmark of the same name.

Recommended citation: Hou, D., Jiang, L., Lin, F., & Yamada, K. D. (2026). CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models. arXiv preprint arXiv:2607.24999.
Read Paper