VRIQ: Benchmarking and Diagnosing the Visual-Reasoning IQ of Vision-Language Models

Abstract

Vision-language model (VLM) errors on visual reasoning may come from missing visual structure, from mistakes after that structure is extracted, or from both. We introduce VRIQ, a benchmark of 1,391 human-verified IQ-style visual reasoning puzzles spanning five psychometric categories, each instantiated in category-matched abstract and natural-image domains. To study perception and reasoning more directly, we also introduce DiagVRIQ, a 209-item diagnostic subset paired with human-verified, rule-neutral textual descriptions. Across 21 VLMs, the 17 non-frontier models reach a 26.8% unweighted category mean on abstract puzzles, while three 2026 frontier models improve substantially but unevenly, with unweighted category means of 41.3–70.9% on abstract puzzles and at most 78.2% on natural-image puzzles; all remain below human performance. On DiagVRIQ, even when their descriptions are judged correct, the strongest models still answer approximately 9–12% of those items incorrectly. Two interventions, ground-truth description augmentation and tool-augmented inference, improve accuracy, with the largest tool gains on spatially demanding categories, but neither eliminates errors.

Benchmarks

• VRIQ — 1,391 puzzles (788 abstract, 603 natural) across 3D, figure rotation, matrix prediction, odd one out, and sequence completion.
• DiagVRIQ — 209 natural puzzles, each paired with a human-verified text description of the image.
• Evaluation code — end-to-end, description-based, and augmentation experiments.

Citation

@article{khezresmaeilzadeh2026vriq,
  title   = {VRIQ: Benchmarking and Diagnosing the Visual-Reasoning
             IQ of Vision-Language Models},
  author  = {Khezresmaeilzadeh, Tina and Zhong, Jike and
             Psounis, Konstantinos},
  journal = {arXiv preprint arXiv:2602.05382},
  year    = {2026}
}