Pylyshyn's Imagery Debate and Spatial AI Architecture

This article explores how cognitive scientist Zenon Pylyshyn’s historic mental imagery debate directly informs the design of modern spatial artificial intelligence. By examining the conflict between propositional descriptions (symbolic code) and depictive arrays (pixel-like grids), we can understand the fundamental trade-offs AI engineers face today when building systems that navigate, understand, and manipulate the physical world. Ultimately, resolving this debate is key to creating efficient, logically sound, and adaptable spatial AI.

To understand spatial AI, we must first understand the two competing methods of representing space that defined the imagery debate of the 1970s and 1980s.

Depictive arrays, championed by Stephen Kosslyn, represent space the way a picture does. In an AI context, this is a voxel grid, a 2D pixel matrix, or a point cloud. Every part of the representation corresponds directly to a part of the physical space, preserving geometry and distance naturally.

Propositional descriptions, championed by Zenon Pylyshyn, represent space using abstract, language-like symbols and logic. Instead of storing an image of a coffee cup on a desk, a propositional system stores the logical statement: ON(Cup, Desk). Pylyshyn argued that human "mental images" are just an illusion, and that the brain actually processes everything using this underlying symbolic code.

Pylyshyn's core critique of depictive arrays is highly relevant to current AI limitations. A raw depictive array—like a 3D map generated by a robot's sensors—is computationally heavy and semantically blind. An AI using only depictive arrays can map the exact distance between two objects, but it does not inherently understand that a "cup" can be picked up while a "desk" cannot. Pylyshyn pointed out that without an underlying propositional framework, a system cannot grasp abstract relationships, causality, or the functional meaning of the space it is looking at.

Furthermore, Pylyshyn introduced the concept of "cognitive penetrability." This means that what a system knows, believes, or intends should be able to directly alter how it processes spatial information. If an AI relies purely on rigid depictive arrays, it cannot easily update its spatial reasoning based on new rules—such as realizing a fragile object requires a wider path of clearance. Propositional descriptions allow an AI to inject logic and physics rules directly into its spatial planning.

On the other hand, relying strictly on Pylyshyn’s propositional descriptions creates its own bottlenecks for spatial AI. Translating a complex, messy 3D room entirely into logical text statements requires massive pre-processing. Worse, it strips away the "free" geometric information that depictive arrays provide. In a depictive grid, if Object A is next to Object B, and Object B is next to Object C, the AI instantly knows the distance between A and C by measuring the grid. In a purely propositional system, the AI must run complex logical deductions to figure out where everything is in relation to everything else, which can slow down real-time navigation.

The imagery debate matters today because it proves that neither approach is sufficient on its own for advanced spatial AI. Purely depictive systems (like standard convolutional neural networks and occupancy grids) struggle with high-level reasoning and long-horizon planning. Purely propositional systems (like traditional symbolic AI or text-only Large Language Models) hallucinate physical impossibilities because they lack a grounded, geometric array to check their logic against.

Modern AI architects are resolving Pylyshyn’s debate by building neuro-symbolic and multimodal systems. By combining depictive arrays for fast, accurate geometric perception with propositional descriptions for abstract reasoning and semantic understanding, spatial AI can both map the physical world and genuinely understand how to interact with it.