Training Stacked RBMs in Early Deep Artificial Intelligence

In the mid-2000s, deep artificial intelligence faced a major hurdle: training multi-layered neural networks was nearly impossible due to vanishing gradients. Researchers overcame this by stacking Restricted Boltzmann Machines (RBMs) to create Deep Belief Networks (DBNs) and Deep Autoencoders. This article explores how these networks were successfully constructed and trained one layer at a time using an unsupervised learning algorithm called Contrastive Divergence, followed by a global fine-tuning phase to revolutionize early deep learning.

Understanding the Restricted Boltzmann Machine

A Restricted Boltzmann Machine (RBM) is a two-layer generative neural network capable of learning a probability distribution over its set of inputs. It consists of a visible layer, which receives the input data, and a hidden layer, which extracts features from that data. The machine is "restricted" because there are no connections between nodes within the same layer; visible nodes only connect to hidden nodes, and vice versa. This symmetric, bipartite structure makes it mathematically simple to infer the states of the hidden nodes given the visible nodes, and to reconstruct the visible nodes given the hidden ones.

Fast Training with Contrastive Divergence

Before RBMs could be stacked, individual RBMs needed an efficient way to learn. Standard training methods required the network to reach thermal equilibrium using prolonged Markov Chain Monte Carlo sampling, which was computationally too slow for practical use. Geoffrey Hinton solved this by introducing Contrastive Divergence (CD).

Instead of waiting for the network to reach equilibrium, Contrastive Divergence approximates the gradient using a shortcut:

  1. The Positive Phase: Input data is clamped to the visible layer, and the probabilities of the hidden layer activations are calculated. The outer product of the visible and hidden vectors is measured.
  2. Reconstruction: The hidden states are used to reconstruct the visible layer, and those reconstructed visible states are then used to activate the hidden layer one more time (a process known as Gibbs sampling, usually done just once in CD-1).
  3. The Negative Phase: The outer product of the reconstructed visible and newly updated hidden vectors is measured.
  4. Weight Update: The weights connecting the layers are updated by taking the difference between the positive phase and the negative phase.

This rapid approximation forced the RBM to adjust its weights so that its reconstructions matched the original input data as closely as possible.

Greedy Layer-by-Layer Stacking

Once a single RBM could be trained quickly via Contrastive Divergence, researchers stacked them to learn complex, hierarchical representations of data. This process is known as greedy layer-by-layer pre-training.

The process begins by training the first RBM directly on the raw input data (such as pixel values) using Contrastive Divergence. Once the weights of this first RBM converge, its parameters are frozen. The hidden layer of this first RBM has now learned to detect low-level features, like edges in an image.

Next, a second RBM is stacked on top of the first. The crucial step is that the hidden layer activations of the first RBM are treated as the "visible" input data for the second RBM. The second RBM is then trained using Contrastive Divergence on these features, rather than the raw data. Once trained, its weights are also frozen. This process is repeated for as many layers as desired. Each subsequent layer learns increasingly abstract representations of the data, moving from edges to shapes, and eventually to complex object parts.

Fine-Tuning the Deep Network

Greedy layer-by-layer training is entirely unsupervised, meaning it uses no data labels. It simply initializes the network's weights into a favorable state so that the deep architecture already understands the underlying structure of the input data.

To make the network useful for specific tasks like classification, the stacked RBMs are converted into a standard deep neural network or a Deep Belief Network. A final output layer representing the target labels is added to the top of the stack. Finally, the entire network undergoes supervised fine-tuning using standard backpropagation and gradient descent. Because the weights were already optimized layer-by-layer via Contrastive Divergence, backpropagation only needs to nudge the weights slightly, completely bypassing the vanishing gradient problem that previously crippled deep artificial intelligence.