How Constitutional AI Automates Critique and Revision

This article provides a comprehensive overview of Constitutional AI, a training framework designed to align artificial intelligence systems using a predefined set of written principles. Instead of relying heavily on human moderators to flag and correct harmful or unhelpful outputs, this method empowers the AI to evaluate and improve its own responses. Below, we explore the core mechanics of the framework: the structure of the AI constitution, the self-critique and revision phase, and the reinforcement learning phase that solidifies safe and helpful AI behavior.

At the core of Constitutional AI is the "constitution" itself. This is a documented list of rules, ethical guidelines, and principles written in plain language by developers. These rules often draw from established human rights declarations, safety guidelines, and company-specific ethical standards. Examples of principles might include instructions to "choose the response that is least harmful" or "avoid generating content that promotes illegal acts." This document serves as the ultimate reference point for the AI during its training process.

The automated training process is divided into two distinct stages: the Supervised Learning Phase and the Reinforcement Learning Phase.

During the Supervised Learning Phase, the AI goes through a continuous cycle of generation, critique, and revision. First, the unaligned AI model is deliberately given prompts designed to provoke harmful, toxic, or biased responses. The AI generates an initial output, which is often flawed. Next, the system feeds this initial output back to the AI, alongside a specific principle selected from the constitution. The AI is asked to critique its own response based on that principle. Once the AI identifies the flaws in its original answer, it is instructed to rewrite the response to resolve the issues. This critique-and-revision loop can be repeated multiple times on the same prompt until the output strictly adheres to the constitution. Finally, the AI is fine-tuned using these newly revised, high-quality responses.

The second stage is the Reinforcement Learning Phase, widely known as Reinforcement Learning from AI Feedback (RLAIF). Traditional AI training relies on humans to compare two AI responses and pick the better one. Constitutional AI automates this step. The fine-tuned AI from the first phase generates a pair of responses to a single prompt. Another AI model, acting as an evaluator, is given the two responses and a principle from the constitution. The evaluator AI determines which of the two responses better aligns with the written rule.

Millions of these automated comparisons are compiled to train a "preference model." This preference model learns to assign a numerical score to any given output based on how well it follows the constitution. The main AI is then trained using reinforcement learning to maximize this score, permanently baking the constitutional principles into its final behavior.

By using written principles to automate critique and revision, Constitutional AI solves major bottlenecks in AI development. It scales the alignment process rapidly, removes the inconsistency of varied human opinions, and spares human workers from the psychological toll of reviewing toxic data. Furthermore, it makes AI safety transparent; if an AI behaves poorly, developers can simply read, adjust, or add to the written constitution to correct the behavior in future training runs.