CogBattery

Benchmark

Tasks

All model responses are text. Memory materials stay visible, and human timing and motor demands are omitted. Human references come from separate published sources, not one common cohort.

Example items

1:G  2:V  3:g

Word: RED
Ink-color label: green

Results

One key measure per paradigm

The measures follow the key-endpoint table of the paper. Invalid task answers count as wrong. There is no overall score. The paradigms measure different things, and a combined human-likeness score would hide that.

Click a column to sort. Cell shading shows how far a model is from the human mean in human standard deviations (across people), full at 3 SD, as in Figure 3 of the paper. It shows distance, not quality, and for several measures a higher value is not better. WAG shows whether a model always placed the same wager, and no human proportion exists for it. Serving shows whether a row comes from the paper's runs or a submission, and which runner backend a submission used. Data in leaderboard.json.

Task correlations in two further human cohorts

Besides SRO and Ball, two public cohorts did several of the tasks. CNP controls did list learning, digit span, and Stroop, and adults in Bradford et al. (2020) did false belief, operation span, and Stroop. Each has one task per paradigm group, so we compare each pair's correlation in humans and in the 54 models. Bradford measures are adjusted for age.

Model correlations exceed human ones in five of six pairs, reliably only for CNP digit span and Stroop. With Stroop interference scores instead of accuracy, no CNP interval excludes zero. Without age adjustment, the Bradford operation span and Stroop interval also excludes zero. Task matches are at task-family level only. Definitions and sensitivity checks are in PAPER_VERSION.md.

Contribute

Submit a model

You run the battery on your own machine and server. The materials are built from the original sources and checked against the ones the paper used.

1. Install the repository and download the sources.
2. Build the paper materials. Each task is checked against the paper's material digest.
3. Run your model with Ollama, as in the paper, or any OpenAI-compatible server. Runs resume after an interruption.
4. See your key measures next to the human references and the paper's 54-model means, with no submission needed.
5. To join the table, pack the run into one small file and open a pull request or an issue with it.
6. We replay your replies through the paper's run loop, check every request, score them with the paper's scorers, and add the row to this table. Scores computed elsewhere are not accepted as is.

pip install -e ".[paper]"
python -m scripts.release.fetch_sources
python -m scripts.release.build_paper_materials
python -m scripts.release.run_model --materials results/paper_materials/manifest.json \
  --model <your-model> --backend ollama --base-url http://localhost:11434
python -m scripts.release.report_model --run results/submissions/<your-model> \
  --materials results/paper_materials/manifest.json
python -m scripts.release.pack_submission pack --run results/submissions/<your-model> \
  --out <your-model>.cogbattery.jsonl.gz

Step-by-step guide, the OpenAI-compatible option, and a no-cost smoke test are in REPRODUCING.md. The CVLT-II word lists are read on your machine from a public UCLA codebook and are not stored in the repository. Results from an OpenAI-compatible server are marked, because its serving differs from the paper's.

Reference

Citation

@article{cogarena2026,
  title   = {CogBattery: Human-Referenced Evaluation of Cognitive Task Behavior and Structure in Language Models},
  journal = {arXiv preprint arXiv:2607.24999},
  year    = {2026}
}