Testing AI Reasoning with Visual Analogical Puzzles

Researchers use Raven's Progressive Matrices and Bongard problems to determine if artificial intelligence models possess true fluid intelligence or merely rely on sophisticated pattern memorization. While modern neural networks excel at recognizing familiar data, they often struggle when forced to deduce abstract rules from novel situations. By testing AI with these two classic visual puzzles, scientists can isolate a model's ability to perform analogical reasoning—the cognitive skill of identifying underlying relationships and applying them to entirely new contexts. This article explores the mechanics of these tests, how they expose the shortcuts AI models take, and how researchers design experiments to verify genuine machine comprehension.

The Distinction Between Memorization and Fluid Intelligence

Fluid intelligence is the capacity to think logically and solve problems in novel situations, independent of acquired knowledge. In contrast, most deep learning models rely on crystallized statistical patterns learned from massive training datasets. When an AI appears to solve a complex problem, it is often matching surface-level features to millions of similar examples it has already seen. To prove an AI can truly reason, researchers must strip away linguistic cues and cultural knowledge, forcing the model to deduce abstract rules from scratch using minimal data.

Using Raven's Progressive Matrices (RPM)

Developed in 1936 for human cognitive testing, a Raven's Progressive Matrix consists of a 3x3 grid of geometric visuals with the bottom-right piece missing. The test-taker must identify the governing rules across the rows and columns—such as changes in shape, shading, rotation, or quantity—and select the correct missing piece from a set of choices.

Researchers use automated generators to create millions of RPM puzzles to train and test AI. However, early experiments revealed a flaw: AI models were achieving superhuman scores without actually reasoning. Instead of analyzing the 3x3 grid, the models analyzed only the multiple-choice answers, exploiting statistical biases in how the incorrect choices were generated.

To counter this memorization shortcut, researchers now use modified RPM frameworks:

Balanced Answer Sets: Algorithms generate choices where the correct answer cannot be guessed without looking at the main puzzle grid.
Out-of-Distribution (OOD) Testing: Researchers train an AI on matrices using specific rules (e.g., rotation and size) and test it on matrices with entirely unseen rules or novel shapes. If the AI only memorized patterns, it fails the OOD test. If it learned how to extract rules, it succeeds.
Neuro-Symbolic Architectures: Scientists use RPMs to test hybrid models that combine neural networks (for visual perception) with symbolic logic systems (for rule deduction), comparing their adaptability against standard deep learning models.

Using Bongard Problems

Introduced by Soviet computer scientist Mikhail Bongard in 1967, a Bongard problem presents two sets of six simple geometric diagrams: Set A on the left, and Set B on the right. All six images in Set A obey a specific abstract rule, while all six images in Set B violate that rule or obey an opposite rule. For example, Set A might contain shapes with a single intersection, while Set B contains shapes with multiple intersections. The test-taker must formulate the rule that separates the two sets.

While RPMs test rule application using large datasets, Bongard problems test "few-shot" concept learning—the ability to deduce a rule from just twelve images. This format makes standard pattern memorization virtually impossible for several reasons:

No Multiple Choice: Because traditional Bongard problems require the solver to articulate the rule rather than pick from a list, AI cannot use elimination heuristics.
Infinite Rule Space: The rules governing Bongard problems rely on topological concepts like "inside versus outside," "convex versus concave," or "clockwise versus counterclockwise." An AI cannot brute-force these concepts through pixel-matching.
Visual Variance: Two drawings in Set A might look completely different at the pixel level (e.g., a tiny jagged triangle and a large smooth square) yet share the same abstract property (e.g., both are closed loops).

How These Tests Expose AI Limitations

When subjected to rigorous RPM and Bongard benchmarks, current AI models consistently fall short of human performance. Deep learning models struggle with what researchers call the "abstraction barrier."

If an AI is trained on thousands of Bongard problems, it may learn to recognize lines and shapes, but it fails to synthesize these perceptions into high-level relational concepts. By modifying a few pixels or changing the thickness of a line, researchers can easily break an AI's ability to solve a puzzle it previously mastered. This fragility proves the model was relying on superficial visual textures rather than understanding the underlying spatial logic.

Ultimately, Raven's Progressive Matrices and Bongard problems serve as a vital reality check for artificial intelligence. Until a model can look at six simple drawings, deduce a never-before-seen topological rule, and apply it to a new context without prior training, AI remains a master of statistical memorization rather than a true reasoning engine.