How Capsule Networks Preserve Spatial Hierarchies in AI

Geoffrey Hinton introduced Capsule Networks (CapsNets) to solve a fundamental flaw in traditional Convolutional Neural Networks (CNNs): the inability to understand spatial relationships between different parts of an object. This article explores how CapsNets preserve hierarchical part-whole relationships in artificial intelligence. It covers the limitations of standard pooling techniques, the introduction of vector-based capsules, the mechanics of dynamic routing by agreement, and how these elements combine to achieve viewpoint invariance.

The Flaw in Traditional CNNs

To understand Capsule Networks, one must first understand the limitation of CNNs. Standard CNNs use a process called "max pooling" to reduce the size of data as it moves through the network. While pooling helps the network recognize if a feature (like an eye or a mouth) is present in an image, it discards the precise spatial data regarding where those features are located relative to one another.

Because of this, a CNN might see two eyes, a nose, and a mouth and classify the image as a face, even if the mouth is on the forehead and the eyes are on the chin. CNNs achieve "translational invariance" (knowing what an object is regardless of where it is), but they fail at preserving the spatial hierarchy between the parts and the whole.

What is a Capsule?

Hinton’s solution was to replace the standard scalar-output neurons of a CNN with "capsules." A capsule is a small group of neurons that outputs a vector (or a matrix in later versions) rather than a single scalar number.

This vector representation allows a capsule to hold two distinct pieces of information simultaneously:

  • Length of the vector: Represents the probability that a specific feature or part exists in the image.
  • Orientation of the vector: Represents the "instantiation parameters" of that part, which include its precise position, size, rotation, thickness, and skew.

By using vectors, the network no longer throws away spatial data. It tracks not just what is in the image, but how it is positioned.

Dynamic Routing by Agreement

Preserving the hierarchy between small parts (like a nose) and the whole object (like a face) requires a mechanism to connect lower-level capsules to higher-level capsules. Hinton replaced max pooling with an algorithm called "Dynamic Routing by Agreement."

Here is how the process preserves part-whole relationships:

  1. Prediction: Lower-level capsules (which detect simple parts like eyes and mouths) use transformation matrices to predict the instantiation parameters (pose and position) of a higher-level capsule (the whole face).
  2. Agreement: If the lower-level capsule for the "nose" and the lower-level capsule for the "mouth" both predict a "face" in the exact same position and orientation, their predictions agree.
  3. Routing: When multiple lower-level capsules agree on the prediction of a higher-level capsule, the connection between them strengthens. The data is dynamically "routed" to that specific higher-level capsule, confirming the presence of the whole object in that exact pose.

If the mouth is on the forehead, its prediction for the position of the face will violently disagree with the prediction made by the eyes. Because there is no agreement, the higher-level "face" capsule will not be activated.

Achieving Viewpoint Equivariance

Through dynamic routing and vector representations, Capsule Networks mirror the human visual system by utilizing inverse graphics. Instead of memorizing an object from thousands of different angles like a CNN must do, a CapsNet understands the 3D geometry of how parts relate to a whole.

This gives the network "equivariance." As an object moves or rotates in the physical world, the orientation of the vectors inside the capsules rotates correspondingly. By locking the spatial relationships of the parts into a strict mathematical hierarchy, Hinton’s Capsule Networks ensured that AI could recognize objects from entirely new viewpoints using a fraction of the training data required by traditional networks.