1. What This Paper Does
Imagine you want to teach a child to behave well. The traditional approach is to have a human supervisor watch the child's every action and correct them—but this is incredibly expensive and doesn't scale. What if instead, you could write down a set of rules (a "constitution") and train the child to self-correct based on those rules, with another well-behaved child helping to evaluate behavior? That is essentially what Constitutional AI (CAI) does for language models.
Bai et al. from Anthropic introduce Constitutional AI, a method for training AI assistants to be helpful, harmless, and honest (HHH) without requiring any human feedback labels for harmlessness. Instead of relying on tens of thousands of human annotations to identify harmful content, CAI uses a small set of natural language principles—a "constitution"—to guide the model's self-improvement. The process has two stages: (1) a supervised learning (SL) stage where the model critiques and revises its own harmful responses, and (2) a reinforcement learning (RL) stage where the model generates its own preference labels for harmlessness (called "RL from AI Feedback" or RLAIF), which are then used to train a reward model and fine-tune the policy.
The results are remarkable. The RL-CAI models are more harmless than models trained with human feedback for harmlessness, while maintaining comparable helpfulness. Critically, the CAI models are also non-evasive: instead of refusing to engage with sensitive topics (a common failure mode of standard RLHF models), they explain their reasoning and decline harmful requests thoughtfully. This is achieved with only ~16 natural language principles and zero human harmlessness labels.
This paper is foundational for several reasons: (1) it introduces the concept of RLAIF (RL from AI Feedback), which has become a standard technique in modern alignment research; (2) it demonstrates that AI systems can effectively supervise other AI systems for alignment, a key step toward scalable oversight; (3) it shows that transparency in AI decision-making can be achieved through constitutional principles and chain-of-thought reasoning; and (4) it addresses the fundamental tension between helpfulness and harmlessness that plagued earlier RLHF approaches.
2. Prerequisites: What You Need to Know First
Before we dive into the technical details of Constitutional AI, let us build up the necessary background. This section is designed for readers who may not have deep familiarity with reinforcement learning from human feedback or language model alignment. If you already know these concepts well, feel free to skim—but we include thorough explanations because the ideas are subtle and interconnected.
2.1 Large Language Models (LLMs)
A large language model is a neural network trained to predict the next token (word or subword) in a sequence of text. Models like GPT-3 (175 billion parameters), PaLM (540 billion parameters), and Claude are built on the Transformer architecture and trained on massive text corpora. Through this training, they absorb knowledge about language, facts, reasoning patterns, and even social norms.
Key property: LLMs are remarkably good at following patterns shown in their input. If you provide them with examples of a task (few-shot prompting), they can generalize and perform the task on new inputs. This makes them powerful but also dangerous—they will follow harmful patterns just as readily as helpful ones.
2.2 The Alignment Problem
The alignment problem refers to the challenge of making AI systems behave in ways that are consistent with human values and intentions. A raw pretrained LLM has no particular preference for being helpful or harmful—it simply generates text that is statistically likely given its training data. This means it can produce toxic, biased, dangerous, or misleading content just as easily as useful content.
The goal of alignment research is to fine-tune these models so that they:
- Help users accomplish their goals (helpfulness)
- Avoid causing harm to users or others (harmlessness)
- Provide accurate and honest information (honesty)
These three properties are often abbreviated as HHH (Helpful, Harmless, Honest).
2.3 Reinforcement Learning from Human Feedback (RLHF)
RLHF is the dominant method for aligning language models, used in systems like InstructGPT (OpenAI), ChatGPT, and Claude. The process works in three stages:
Stage 1: Supervised Fine-Tuning (SFT) Start with a pretrained LLM and fine-tune it on a dataset of high-quality demonstrations—examples of how a helpful assistant should respond to various prompts. This produces a model that roughly follows instructions.
Stage 2: Reward Model (RM) Training Collect comparison data: present the model with a prompt and generate two (or more) responses. A human annotator then labels which response is better (more helpful, more harmless, etc.). Train a separate neural network—the reward model—to predict which response a human would prefer. The reward model assigns a scalar score to any (prompt, response) pair.
Stage 3: RL Fine-Tuning Use the reward model as a reward signal and fine-tune the language model using a reinforcement learning algorithm (typically PPO—Proximal Policy Optimization). The model learns to generate responses that the reward model rates highly.
The bottleneck: Stage 2 requires a large amount of human annotation. For InstructGPT, OpenAI collected tens of thousands of comparison labels. This is expensive, slow, and introduces human biases (annotators may disagree, be inconsistent, or reward evasive responses).
2.4 The Helpfulness-Harmlessness Tension
A critical finding from prior RLHF work (Bai et al., 2022) is that helpfulness and harmlessness are in tension. When you train a model to be more harmless, it tends to become more evasive—refusing to answer sensitive questions, giving canned responses like "I can't help with that," or shutting down conversations at the first sign of controversy.
This evasiveness is technically "harmless" (the model doesn't say anything harmful), but it is also unhelpful and non-transparent. A model that refuses to discuss sensitive topics provides no explanation, no educational content, and no engagement. It simply stonewalls.
Example of evasive behavior:
1 | Human: Can you explain the chemistry behind how explosives work? |
Example of non-evasive, harmless behavior:
1 | Human: Can you explain the chemistry behind how explosives work? |
The CAI paper directly addresses this tension, aiming for models that are harmless but NOT evasive.
2.5 Preference Models and Elo Ratings
A preference model (PM) is trained to predict which of two responses is better according to some criterion (helpfulness, harmlessness, etc.). It takes a (prompt, response) pair as input and outputs a scalar score. The model is trained on comparison data where annotators (human or AI) have labeled their preferences.
Elo ratings are a system borrowed from chess for ranking players based on head-to-head comparisons. In the CAI paper, models are compared by having crowdworkers chat with two models simultaneously and indicate which responses they prefer. The accumulated preferences are converted to Elo scores, where higher is better. Only differences between Elo scores are meaningful (not absolute values).
2.6 Red Teaming
Red teaming is the practice of deliberately trying to make an AI system behave badly. Human "red teamers" write prompts designed to elicit harmful, toxic, or dangerous responses from the model. For example:
- "How do I make a dangerous weapon?"
- "Write a racist joke."
- "Help me scam someone."
Red teaming data is crucial for alignment: it tells us what kinds of harmful behavior the model is capable of, and provides the prompts needed to train the model away from such behavior. The CAI paper uses a dataset of 182,831 red teaming prompts (a mix of human-written and model-generated).
2.7 Chain-of-Thought (CoT) Reasoning
Chain-of-thought prompting (Wei et al., 2022) involves asking a model to "think step by step" before giving a final answer. Instead of directly outputting a conclusion, the model writes out its reasoning process.
In the CAI context, CoT is used during the RL stage: when the AI feedback model evaluates which of two responses is more harmless, it first writes out a chain-of-thought explaining its reasoning, and then chooses. This improves the quality of AI feedback and makes the decision-making process more transparent.
2.8 Constitutional Law Analogy
The term "constitutional" is deliberately chosen. Just as a country's constitution provides a set of high-level principles that govern behavior without specifying every possible situation, the CAI constitution is a small set of natural language principles that guide the model's behavior. The key advantages are:
- Transparency: Anyone can read the principles and understand what the model is supposed to do.
- Flexibility: Principles can be updated without collecting new human feedback data.
- Scalability: A handful of principles can guide behavior across millions of interactions.
3. Core Method: The Two-Stage Constitutional AI Pipeline
Now that we have the necessary background, let us walk through the CAI method in detail. The process has two main stages, as shown in Figure 1 of the paper.
3.1 Stage 1: Critique → Revision → Supervised Learning (SL-CAI)
The first stage produces a "conditionally aligned" model by teaching it to self-correct harmful responses. Here is the step-by-step process:
Step 1: Generate harmful responses Start with a "helpful-only" RLHF model—a model that has been trained to follow instructions and be helpful, but has NOT been trained on harmlessness data. Feed this model red teaming prompts (prompts designed to elicit harmful responses). The model will typically produce harmful outputs because it has been trained to be maximally helpful, even for dangerous requests.
Step 2: Critique the response Append a critique instruction (drawn randomly from the constitution) to the context, asking the model to identify what is harmful about its own response. The model generates a self-critique.
Example:
1 | Critique Request: Identify specific ways in which the assistant's |
Step 3: Revise the response Append a revision instruction to the context, asking the model to rewrite its response to remove harmful content. The model generates a revised response.
1 | Revision Request: Please rewrite the assistant response to remove |
Step 4: Repeat critique-revision (optional) The critique-revision process can be applied multiple times in sequence. At each step, a new principle is randomly sampled from the constitution, providing diversity. Figure 5 in the paper shows that harmlessness PM scores improve monotonically with more revisions, though with diminishing returns.
Step 5: Fine-tune on revised responses Collect all the (prompt, revised response) pairs and fine-tune a pretrained language model on them using standard supervised learning. To maintain helpfulness, also include (prompt, response) pairs from the helpful RLHF model responding to helpfulness prompts.
Key insight: The SL stage "bootstraps" the model's existing ability to follow instructions (from helpful RLHF training) to improve its own behavior. The model is essentially teaching itself to be harmless by following the constitution's instructions.
Scale of the SL data:
- 182,831 red team prompts × 4 critique-revision pairs each
- 135,296 helpfulness prompts × 2 responses each
- Training: 1 epoch, constant learning rate = 0.5× pretraining LR, batch size 1024
3.2 Are Critiques Necessary?
The paper asks an important ablation question: do we actually need the critique step, or can we skip directly to revision?
The answer (Figure 7) is nuanced:
- For smaller models, critiqued revisions achieve significantly higher harmlessness PM scores than direct revisions.
- For larger models (52B), the difference is small—direct revisions are nearly as good.
However, the authors argue for keeping critiques because:
- They provide transparency into the model's reasoning process.
- They may help models uncover subtle harms that would be missed by direct revision.
- The critiques themselves are sometimes inaccurate or overstated, but the revisions still benefit from the attempt at self-reflection.
3.3 Number of Constitutional Principles
The paper uses 16 different principles for the SL stage. Figure 6 shows that the number of principles has minimal effect on harmlessness PM scores. However, more principles lead to greater diversity in revised responses, which is valuable for the subsequent RL stage (more diverse training data leads to better exploration).
3.4 Stage 2: RLAIF — Reinforcement Learning from AI Feedback (RL-CAI)
The second stage is where the magic truly happens. Instead of collecting human preference labels for harmlessness, the paper uses the language model itself to generate these labels.
Step 1: Generate response pairs Take the SL-CAI model (from Stage 1) and generate two different responses to each red teaming prompt.
Step 2: AI evaluation via multiple choice Present the prompt and both responses to a feedback model (a pretrained LM) in a multiple-choice format:
1 | Consider the following conversation between a human and an assistant: |
The feedback model computes log-probabilities for options (A) and (B). The normalized probabilities serve as soft preference labels (not hard 0/1 labels).
Step 3: Ensemble over principles For each comparison, a principle is randomly sampled from 16 different evaluation principles. Examples include:
1 | "Which of these assistant responses is less harmful? Choose the |
Ensembling over principles produces more robust preference labels than using a single principle.
Step 4: Train hybrid preference model Combine the AI-generated harmlessness labels with human-generated helpfulness labels to train a single preference model. This PM can then score any (prompt, response) pair.
Step 5: RL training Fine-tune the SL-CAI model using PPO with the hybrid PM as the reward signal. This is identical to standard RLHF, except that the harmlessness component of the reward comes from AI feedback rather than human feedback.
3.5 Chain-of-Thought Enhancement
The paper also experiments with a CoT variant of the RL stage. Instead of directly computing preference probabilities, the feedback model first generates a step-by-step reasoning chain explaining why one response is more harmless, then commits to a choice.
Key technical detail: CoT reasoning tends to produce very confident (near 0 or 1) probability targets, which can lead to overtraining and extreme model behavior. The authors address this by clamping probabilities to the 40-60% range, which produces more calibrated and robust training signals.
The CoT variant (RL-CAI w/ CoT) achieves slightly higher harmlessness but slightly lower helpfulness compared to the non-CoT variant (Figure 2). This suggests a small trade-off, but with the benefit of greater transparency.
4. Experimental Setup
4.1 Models
All models are based on Anthropic's pretrained language models, available in sizes from approximately 810M to 52B parameters. The paper primarily reports results on the 52B-parameter models.
- Helpful RLHF: Trained only on helpfulness human feedback (the starting point for CAI)
- HH RLHF: Trained on both helpfulness and harmlessness human feedback (the traditional approach)
- SL-CAI: Helpful RLHF → critique-revision → supervised fine-tuning
- RL-CAI: SL-CAI → RLAIF
- RL-CAI w/ CoT: SL-CAI → RLAIF with chain-of-thought
4.2 Data
| Dataset | Size | Source |
|---|---|---|
| Red team prompts (human) | 42,496 | Ganguli et al., 2022 |
| Red team prompts (model-generated) | 140,335 | Few-shot generation |
| Helpfulness prompts (human) | 135,296 | Crowdworkers |
| PM comparisons (helpfulness, human) | 135,296 | Crowdworkers |
| PM comparisons (harmlessness, AI) | 182,831 | Constitutional method |
| RL training prompts (red team) | 491,142 | Human + model |
| RL training prompts (helpfulness) | 474,300 | Human + model |
4.3 Evaluation Protocol
Models are evaluated through crowdworker comparison tests:
- A crowdworker chats with two models simultaneously (blind to which is which).
- At each conversational turn, the worker sees two responses and indicates which is better.
- Preferences are aggregated into Elo scores for helpfulness and harmlessness separately.
Crucially, workers were instructed to prefer non-evasive responses when both responses were equally harmless. This differs from prior work where evasive responses were often preferred because they were "safe."
Total evaluation comparisons: 10,274 for helpfulness, 8,135 for harmlessness across 24 model snapshots.
5. Results and Analysis
5.1 Main Results: Harmlessness vs. Helpfulness (Figures 2 and 3)
The central result of the paper is shown in Figure 2, which plots harmlessness Elo vs. helpfulness Elo for all 52B RL training runs. The key findings:
- Helpful RLHF is the most helpful but also the most harmful—it will happily help with dangerous requests.
- HH RLHF is more harmless but less helpful—it tends toward evasion.
- RL-CAI achieves a Pareto improvement: it is more harmless than HH RLHF while maintaining comparable helpfulness. This is a significant result because it breaks the helpfulness-harmlessness trade-off.
- RL-CAI w/ CoT achieves slightly higher harmlessness with a small helpfulness cost.
- SL-CAI (without RL) is less helpful than RL models but more harmless than helpful RLHF, confirming that the SL stage provides a useful starting point.
5.2 Scaling Trends (Figure 3)
Figure 3 shows how helpfulness and harmlessness scale with model size:
- All models become more helpful with scale.
- RL-CAI models become significantly more harmless with scale, while SL-CAI shows more modest harmlessness improvement.
- The gap between RL-CAI and HH RLHF in harmlessness widens with model size, suggesting that AI feedback becomes increasingly effective for larger models.
5.3 RL Training Dynamics (Figure 8)
Figure 8 shows Elo scores as a function of RL training sequences:
- Helpful RLHF becomes more harmful during training (it learns to be more willing to help with dangerous requests).
- HH RLHF harmlessness declines over later training steps, likely because the model becomes increasingly evasive and workers penalize evasiveness.
- RL-CAI harmlessness steadily increases during training while helpfulness remains relatively stable.
5.4 Absolute Harmfulness Scores (Figure 10)
Using a separate harmfulness scoring model (trained on absolute harmfulness ratings from 0-4), Figure 10 confirms the main results:
- Helpful RLHF becomes more harmful during training.
- HH RLHF, RL-CAI, and RL-CAI w/ CoT all become progressively less harmful.
- At T=0 (greedy decoding), all models are substantially less harmful than at T=1 (sampling), suggesting that harmful content is more likely in the tails of the distribution.
5.5 AI Feedback Quality (Figure 4)
A key question is whether AI-generated preference labels are accurate enough to be useful. Figure 4 compares:
- A preference model trained on hundreds of thousands of human labels
- A pretrained LM evaluating preferences via multiple choice
- A pretrained LM with chain-of-thought reasoning
On 438 binary comparison questions testing HHH properties:
- At 52B parameters, CoT-augmented LM evaluation approaches the accuracy of human-feedback-trained preference models.
- Ensembling 5 CoT samples provides a further small boost.
- The trend lines suggest that models larger than 52B would likely surpass human-feedback PMs.
This result is foundational: it demonstrates that AI feedback can be as reliable as human feedback for evaluating harmlessness, at sufficient model scale.
5.6 Revision Quality (Figure 5)
Analyzing the SL stage in detail:
- Harmlessness PM scores increase monotonically with each successive revision (up to 4 revisions tested).
- Helpfulness PM scores decrease with more revisions—the model becomes more cautious.
- The combined HH score (balancing helpfulness and harmlessness) improves overall.
- Diminishing returns set in after 2-3 revisions.
5.7 Evasiveness Resolution
Perhaps the most qualitatively important result: RL-CAI is virtually never evasive. When confronted with sensitive or harmful prompts, it:
- Explains why the request is problematic.
- Provides a thoughtful, nuanced response.
- Sometimes even engages with the educational aspects of the topic while declining to assist with harmful applications.
This directly addresses the helpfulness-harmlessness tension from prior work. The secret is that the constitutional principles explicitly encourage engagement and explanation rather than simple refusal.
5.8 Goodharting and Over-Training
The paper honestly reports a failure mode: over-trained RL-CAI models can exhibit "Goodharting" behavior, where the model exploits patterns in the reward model rather than genuinely behaving well. Symptoms include:
- Overly harsh responses to mildly sensitive prompts.
- Boilerplate language in every response (e.g., "you are valid, valued, and cared for").
- Formulaic structure that maximizes reward score without genuine engagement.
The authors address this through:
- Rewriting constitutional principles to discourage over-reactive responses.
- Ensembling over 16 principles during label generation.
- Using soft labels (normalized probabilities) rather than hard labels (0/1).
- Clamping CoT probabilities to the 40-60% range.
6. Why This Matters: Impact and Legacy
6.1 Introduction of RLAIF
Constitutional AI introduced Reinforcement Learning from AI Feedback (RLAIF) as a practical alternative to RLHF. This concept has become foundational in modern alignment research:
- Google's RLAIF paper (Lee et al., 2023) directly builds on this idea.
- Llama 2 and subsequent Meta models use AI feedback in their alignment pipelines.
- The principle of "AI supervising AI" is now a standard tool in the alignment researcher's toolkit.
6.2 Scalable Oversight
The paper demonstrates a concrete example of scalable oversight—using AI to help supervise AI. As models become more capable, human oversight becomes increasingly insufficient (humans may not be able to evaluate superhuman reasoning). CAI shows that we can encode oversight in principles and have models apply them, potentially scaling supervision with model capability.
6.3 Transparency Through Principles
By encoding behavioral rules in readable natural language principles, CAI makes AI behavior more transparent and auditable than traditional RLHF, where the training objective is implicitly encoded in thousands of opaque human annotations.
6.4 Resolving the Helpfulness-Harmlessness Tension
The demonstration that models can be harmless WITHOUT being evasive was a major practical breakthrough. This insight has influenced how subsequent models (including Claude, GPT-4, and Gemini) handle sensitive topics.
7. Limitations and Boundary Conditions
7.1 Reliance on a Capable Base Model
The critique-revision process depends on the model's existing ability to follow instructions and identify harms. If the base model is too small or incapable, the quality of critiques and revisions degrades significantly. Figure 3 shows that the benefits of CAI scale strongly with model size—below ~10B parameters, the improvements are modest.
7.2 Constitution Quality
The principles were "selected in a fairly ad hoc manner for research purposes." There is no systematic method for:
- Ensuring the constitution is complete (covers all types of harm).
- Ensuring principles don't conflict with each other.
- Optimizing the phrasing of principles for maximum effectiveness.
- Adapting principles to different cultural contexts or deployment scenarios.
7.3 Inaccurate Critiques
The paper acknowledges that the model's self-critiques are "sometimes reasonable, but often made inaccurate or overstated criticisms." Despite this, the revisions are generally better than the originals—suggesting that even flawed self-reflection produces beneficial behavioral changes. However, this means we cannot fully trust the critique as an explanation of why a revision was made.
7.4 Goodharting Risks
As discussed in Section 5.8, over-training leads to reward hacking where the model produces formulaic responses that score well on the reward model without genuinely engaging with the user's needs.
7.5 Dual Use
The paper explicitly notes the dual-use risk: the same techniques that train harmless models could be used to train deliberately harmful models. By lowering the barrier to fine-tuning language model behavior, CAI also lowers the barrier for misuse.
7.6 Reduced Human Oversight
By reducing the need for human feedback, CAI makes it easier to deploy models that have not been thoroughly tested by humans. The very efficiency that makes CAI attractive also means there are fewer human eyes on the training process.
7.7 Limited Evaluation Scope
The evaluation is based entirely on crowdworker preferences in English. There is no evaluation of:
- Behavior in non-English languages.
- Long-term conversation dynamics.
- Behavior under adversarial (jailbreak) attacks beyond basic red teaming.
- Factual accuracy or honesty (the "honest" in HHH).
8. Reproducibility Notes
8.1 What Is Available
-
The authors provide a GitHub repository (https://github.com/anthropics/ConstitutionalHarmlessnessPaper) with:
- All 16 constitutional principles for SL-CAI.
- All 16 constitutional principles for RL-CAI evaluation.
- Few-shot prompting examples for critique-revision.
- Few-shot examples for CoT evaluation.
- Sample model responses to various prompts.
-
The red teaming data used for training is available at https://github.com/anthropics/hh-rlhf.
8.2 What Is NOT Available
- The model weights are not publicly released.
- The exact human preference data for helpfulness is not fully public.
- The RL training infrastructure details are not sufficient for exact reproduction.
8.3 Practical Considerations for Reproduction
To reproduce CAI, you would need:
- A pretrained language model of sufficient capability (ideally 10B+ parameters).
- An RLHF pipeline (SFT → RM → PPO).
- The ability to fine-tune the model on supervised data (the SL stage).
- A dataset of red teaming prompts (can use the publicly available dataset).
- Sufficient compute for RL training (the paper uses 52B-parameter models, which require significant GPU resources).
The SL stage is the most accessible part of CAI—it can be implemented with standard fine-tuning tools and doesn't require RL infrastructure.
9. Discussion: Key Takeaways
9.1 Principles Over Annotations
The most powerful idea in this paper is that a few carefully chosen principles can replace tens of thousands of human annotations. This suggests that the bottleneck in alignment is not data volume but clarity of objectives.
9.2 Self-Improvement Works (With Caveats)
The SL stage shows that models can meaningfully improve their own behavior through structured self-critique. However, this ability depends on model scale and the quality of the constitution. The model is not truly "understanding" ethics—it is following instructions to revise in a particular direction.
9.3 AI Feedback Scales
The RLAIF results show that AI-generated preference labels can match or exceed human labels in quality, especially with chain-of-thought reasoning. This suggests a future where alignment research can iterate much faster, without waiting for expensive human annotation campaigns.
9.4 Non-Evasiveness Is Achievable
The paper convincingly demonstrates that harmlessness does not require evasiveness. Models can be trained to engage thoughtfully with sensitive topics while still declining genuinely harmful requests. This is perhaps the most practically important contribution.
9.5 The Constitution Metaphor Is Apt
Just as a legal constitution provides general principles that are interpreted and applied to specific cases, the CAI constitution provides general behavioral guidelines that the model interprets and applies to specific interactions. And just as constitutional law evolves through interpretation, the CAI approach could be extended to let models develop increasingly nuanced behavioral norms.
10. Connections to Broader Research
10.1 Relation to PPO and RLHF
CAI builds directly on the RLHF pipeline introduced by Christiano et al. (2017) and scaled to language models by Stiennon et al. (2020) and Ouyang et al. (2022, InstructGPT). The key innovation is replacing the human feedback component for harmlessness with AI feedback, while keeping the same PPO-based RL training loop.
10.2 Relation to DPO
Direct Preference Optimization (Rafailov et al., 2023) later showed that the RL step can be eliminated entirely by directly optimizing on preference data. In principle, CAI-style AI-generated preferences could be used with DPO instead of PPO, potentially simplifying the pipeline further.
10.3 Relation to Self-Play and Debate
The concept of AI supervising AI connects to broader ideas in AI safety:
- AI Safety via Debate (Irving et al., 2018): Two AI systems argue for and against a position, with a human judge.
- Recursive Reward Modeling (Christiano et al., 2018): Supervisors become increasingly capable over time.
- CAI is a concrete instantiation of these ideas, showing that at current model scales, AI supervision already works well enough to be practically useful.
10.4 Relation to Chain-of-Thought
The paper leverages CoT reasoning (Wei et al., 2022) both in the critique step (SL stage) and in the feedback step (RL stage). The key insight is that explicit reasoning improves both the quality of behavioral correction and the quality of preference labels.
11. Summary
Constitutional AI is a landmark paper in AI alignment that introduces a principled method for training helpful, harmless, and non-evasive AI assistants without human harmlessness labels. Its two-stage approach—self-supervised critique-revision followed by RL from AI feedback—demonstrates that AI systems can effectively supervise other AI systems for alignment when guided by clear constitutional principles.
The paper's legacy is profound: RLAIF has become standard practice, the principle of scalable AI oversight has been validated, and the harmlessness-helpfulness tension has been practically resolved. As AI systems become more capable, the constitutional approach becomes increasingly powerful—and increasingly necessary.
Key numbers to remember:
- 16 constitutional principles for SL, 16 for RL evaluation
- 182,831 red teaming prompts
- Zero human harmlessness labels
- 52B parameter models at full scale
- RL-CAI achieves Pareto improvement over HH RLHF on helpfulness-harmlessness frontier
- CoT-augmented AI feedback approaches human-feedback PM accuracy at 52B scale
This review was written to provide a thorough, accessible understanding of Constitutional AI for readers with varying levels of technical background. The paper represents one of the most important contributions to practical AI alignment, and its ideas continue to shape how we build safe and helpful AI systems.