How LLMs Are Created — Pre-training, Post-training, and RL

Why This Matters

Every AI assistant you've used — ChatGPT, Claude, Gemini, DeepSeek — emerged from the exact same two-phase pipeline. A base model reads a significant fraction of the internet and learns to predict the next token. Then post-training sculpts that raw capability into a helpful, harmless assistant that follows instructions, refuses harmful requests, and — in the latest models — thinks through multi-step reasoning problems before answering.

What makes this worth understanding deeply is that the competitive landscape has inverted. Through 2023, the differentiator was who could pre-train the biggest model. By 2025, the frontier had shifted: post-training is where the majority of usable capability is created. Liquid AI's benchmarks show that post-training alone improves performance by 20–40% — gains that would require orders of magnitude more pre-training compute to match. The recipe for a frontier model is no longer "more GPUs, more data." It's "better SFT curation, smarter RL, and iterative preference optimization."

This article walks through every stage — the architecture, the data, the loss functions, and the mathematical machinery that turns a distribution over internet text into a reasoning agent.

Prerequisites

What you should already know

• Transformer decoder architecture: masked self-attention, feed-forward networks, residual connections, layer normalization
• Neural network training fundamentals: forward pass, backward pass, loss functions, gradient descent
• Basic probability: softmax, cross-entropy, KL divergence, log-probabilities
• Familiarity with what an LLM is from the user's perspective (you've used ChatGPT or Claude)

Core Idea

Pre-training and post-training serve two fundamentally different purposes, and understanding their separation is the key to understanding modern AI.

Pre-training is distribution learning. The model reads trillions of tokens from the internet and learns one task: given the previous tokens, predict the next one. This sounds simple — almost trivial — but predicting the next token across the entire internet forces the model to internalize grammar, facts, reasoning patterns, code structure, and cultural knowledge. The output is a base model: terrifyingly knowledgeable, capable of completing any text, and completely useless as an assistant. If you prompt it with a question, it will statistically complete that question as if it were a forum post — maybe with an answer, maybe with another question, maybe with a list of unrelated follow-ups.

Post-training is distribution steering. It takes the base model and reshapes its output distribution toward specific desirable behaviors: following instructions, expressing uncertainty, refusing harmful requests, and — in the newest generation — producing extended chains of reasoning before answering. Post-training itself has multiple stages, each with its own objective and data strategy. Supervised Fine-Tuning (SFT) teaches the format of being an assistant. Preference alignment (RLHF, DPO) teaches the model which responses are better when both are plausible. And reinforcement learning with verifiable rewards (GRPO, RLVR) teaches the model to solve problems with objectively correct answers — math, code, logic — through extended reasoning traces.

The Newly Possible lens: before post-training was understood as a multi-stage pipeline, models were knowledgeable but unusable. Post-training unlocked the transformation from "text completer" to "assistant" — and later, from "assistant" to "reasoning agent." Each stage in the pipeline enables a capability that was previously impossible.

How It Actually Works

Phase 1: Pre-training — Learning the Distribution of Human Text

1.1 The Architecture: Why Decoder-Only Won

Every major LLM since GPT-2 uses a decoder-only Transformer. The architecture stacks N identical blocks, each containing:

• Masked multi-head self-attention — computes how much each token should attend to every previous token (causal mask prevents looking ahead)
• Feed-forward network — typically with SwiGLU activation (a gated variant: $(\mathbf{x} \mathbf{W}_1 \odot \text{silu}(\mathbf{x} \mathbf{W}_2)) \mathbf{W}_3$), applying a position-wise transformation
• RMSNorm (Root Mean Square Layer Normalization) — a simpler, faster alternative to LayerNorm used in Llama, DeepSeek, and most modern architectures
• Residual connections around both sublayers

The full forward path: token embeddings → add positional encoding (RoPE, Rotary Position Embedding, now near-universal) → N × (RMSNorm → masked attention → residual add → RMSNorm → FFN → residual add) → final RMSNorm → unembedding projection → softmax over vocabulary.

Key architectural decisions that compound at scale:

• Pre-norm over post-norm: normalizing before each sublayer (not after) stabilizes training at depth. Every model since GPT-3 uses this.
• Grouped-query attention (GQA): instead of every attention head having its own key/value projections, heads share them in groups. Llama 2 70B uses 8 key-value heads for 64 query heads — same quality, dramatically less KV-cache memory at inference.
• Sliding window attention (Mistral): each token only attends to the previous W tokens, making attention cost O(W) instead of O(n²) for long sequences.

1.2 The Data Pipeline: From Common Crawl to Training Corpus

The internet is the raw material. The pipeline that converts it into training data is where the real engineering happens.

Collection. The starting point is Common Crawl — a nonprofit that crawls the web monthly and releases petabytes of raw HTML. But raw Common Crawl is mostly junk: navigation bars, ads, duplicate pages, auto-generated spam, boilerplate. Less than 10% survives filtering.

Filtering. The multi-stage pipeline:

1. Language detection — remove non-target languages (fastText classifier)
2. Heuristic filtering — remove documents that are too short (< 100 words), have excessive punctuation repetition, abnormal word-length distributions, or high proportions of boilerplate (menus, copyright footers)
3. Perplexity filtering — train a small n-gram model on high-quality reference text (Wikipedia, books), score every document, discard those with anomalously high perplexity. This catches machine-generated spam and incoherent text.
4. Deduplication — exact dedup (hash matching) removes literal copies. Fuzzy dedup (MinHash LSH) catches near-duplicates: pages that differ by a few words, boilerplate variations, mirrored content. One study found that deduplicating Common Crawl reduces corpus size by ~50% while improving downstream model quality — duplicated data wastes compute and causes memorization.
5. Quality classifiers — models trained to distinguish "high quality" (Wikipedia, textbooks) from "low quality" (forum spam, SEO pages) score each document. Documents below a threshold are discarded.

Domain mixing. The filtered corpus is blended from multiple sources, and the blend ratios are a tuned hyperparameter. A typical modern mix:

Source	Share	Why it matters
Filtered web (Common Crawl)	65–75%	Breadth, world knowledge, diversity
Code (GitHub, Stack Overflow)	10–20%	Reasoning, structure, formal patterns
Academic text (ArXiv, PubMed)	3–5%	Technical depth, precise language
Books (fiction + nonfiction)	3–5%	Long-form coherence, narrative structure
Wikipedia, curated sources	2–3%	Factual grounding, high signal-to-noise

Code deserves special mention. Nearly every lab now reports that including code in pre-training improves reasoning on non-code tasks — math, logic, even natural language inference. The hypothesis: code is formal, unambiguous, and structured in ways that natural language isn't. Training on it forces the model to develop attention patterns and internal representations that transfer to other reasoning domains.

Tokenization. The tokenizer is trained once on a representative sample of the corpus, then frozen. It's never updated — changing the tokenizer means retraining the entire model from scratch.

Modern LLMs use Byte-Pair Encoding (BPE) with byte-level fallback: start with individual bytes as the vocabulary, iteratively merge the most frequent adjacent pairs, stop when the vocabulary reaches a target size (typically 32K–256K tokens). The byte-level fallback means the tokenizer can encode any Unicode string, even characters that never appeared in the training data — critical for handling code, math notation, and multilingual text.

Vocabulary size is a balancing act. A larger vocabulary means shorter sequences (fewer tokens to process) but larger embedding matrices (vocab_size × d_model parameters) and less sharing between similar words. Llama 2 uses 32K. GPT-4 uses ~100K. DeepSeek-V2 uses 128K.

1.3 The Training Objective: Why Next-Token Prediction Works

The objective is deceptively simple. Given a sequence of tokens $x_1, x_2, ..., x_T$, the model computes:

$\mathcal{L}_{LM} = -\frac{1}{T} \sum_{t=1}^{T} \log P_\theta(x_t \mid x_1, \ldots, x_{t-1})$

At each position $t$, the model predicts a probability distribution over the entire vocabulary for the next token, and the loss is the negative log-probability of the actual next token. Minimize this, and the model learns to assign high probability to tokens that actually appear in real text.

Why does this work? Because language is compressible. The next token is not uniformly random — it's highly constrained by the preceding context. To predict it well, the model must learn everything that constrains it: grammar (what tokens can legally follow?), facts (Paris is the capital of _____), reasoning patterns (if X then Y, therefore _____), stylistic conventions, cultural references, and code structure. Every piece of knowledge that makes one token more likely than another gets baked into the model's weights.

Teacher forcing is the critical training detail: during training, the model always receives the ground-truth preceding tokens as input, never its own predictions. This avoids the compounding error problem — where an early mistake causes all subsequent predictions to be made from out-of-distribution context. It makes training stable and parallelizable across the entire sequence.

1.4 Scaling Laws: From Kaplan to Chinchilla and Beyond

The single most important empirical finding in LLM research: loss scales as a predictable power law with compute, model size, and data.

Kaplan et al. (OpenAI, 2020). Trained a family of models across six orders of magnitude of compute. The headline: for a given compute budget, you should increase model size faster than data size. The optimal ratio was approximately 1.7 tokens per parameter. This drove the design of GPT-3 (175B parameters, 300B tokens).

Chinchilla (Hoffmann et al., DeepMind, 2022). The correction that reshaped the entire field. Kaplan had made a methodological error: they varied model size while keeping training tokens fixed, which biased the results toward larger models. DeepMind trained 400+ models, varying both model size and training tokens, and found the true optimal ratio: ~20 tokens per parameter.

What this meant in practice: GPT-3 was massively undertrained. To be compute-optimal:

• Keep 300B tokens → the model should be only ~15B parameters (11× smaller)
• Keep 175B parameters → you need ~3.5T tokens (11× more data)

The Chinchilla paper demonstrated this experimentally: their 70B model trained on 1.4T tokens matched or outperformed Gopher (280B) on almost every benchmark — with 4× fewer parameters and 4× less compute per inference.

Beyond Chinchilla (2023–2025). Newer work has pushed the optimal ratio even higher:

Model	Parameters	Training Tokens	Ratio
GPT-3 (Kaplan)	175B	300B	1.7:1
Chinchilla	70B	1.4T	20:1
Llama 2	70B	2T	28:1
DeepSeek-V2	236B (21B active)	8.1T	~34:1
Llama 3	405B	15T+	~37:1
Llama 3.1 8B	8B	15T	1,875:1

Llama 3.1's 8B model at 1,875:1 is the extreme data point — and even then, Meta reported the model had not converged. Andrej Karpathy's assessment: "LLMs are significantly undertrained by a factor of maybe 100–1000× or more." The "data wall" — running out of unique high-quality text — may be further away than we thought, or we may need to get comfortable with repeated data.

1.5 Training Infrastructure at Frontier Scale

Training a frontier model is as much a systems problem as a machine learning one. The scale is staggering:

• GPT-4 (2023): estimated ~25,000 NVIDIA A100 GPUs for 90–100 days. Compute cost: $78M–$100M according to the Stanford AI Index Report 2025.
• Gemini Ultra 1.0 (2024): estimated $192M in compute alone.
• A 1T-parameter model at Chinchilla-optimal scale would need ~20T tokens and roughly 10²⁵ FLOPs — a $500M+ training run at current cloud prices.

The infrastructure that makes this possible:

3D parallelism. No single GPU can hold a frontier model. The model and data are split across thousands of GPUs using three complementary strategies:

• Data parallelism: replicate the entire model on each GPU, split the batch. Each GPU processes a different subset, gradients are averaged (all-reduce) after each step.
• Tensor parallelism: split individual layers across GPUs. A single attention or FFN matrix multiplication is partitioned so each GPU holds a slice. Requires high-bandwidth interconnects (NVLink) — the GPUs communicate on every forward/backward pass.
• Pipeline parallelism: split the model by layers. GPU 1 handles layers 1–8, GPU 2 handles layers 9–16, etc. Micro-batches are pipelined to keep all GPUs busy. The challenge: "bubbles" where GPUs idle waiting for upstream micro-batches.

Mixed precision training. Forward and backward passes use bf16 (16-bit brain floating point) — half the memory of fp32, sufficient dynamic range. Optimizer states (momentum, variance) stay in fp32 to preserve numerical precision. This is now standard: every major framework defaults to bf16.

Gradient accumulation. To simulate a large global batch size without running out of GPU memory: process several micro-batches sequentially, accumulate gradients without updating weights, then apply one update. If each micro-batch is 4 sequences and you accumulate 64 times, your effective batch size is 256.

Activation checkpointing. During the forward pass, only store activations at a subset of layers. During the backward pass, recompute the missing activations on the fly. Trades ~30% more compute for dramatically less memory.

Resilience. Runs at this scale fail — hardware errors, network partitions, silent data corruption. Training frameworks support checkpointing (save model state every N steps) and automatic restart. A single GPU failure in a 25,000-GPU cluster is a statistical certainty every few hours.

1.6 What Pre-training Produces

The output of pre-training is a base model. It has never seen an instruction format. It doesn't know it's an assistant. If you give it a question, it may answer, ask a follow-up, write a poem, or produce a list of search results — it's statistically completing text, not responding to a user.

But base models possess remarkable emergent capabilities:

• In-context learning: provide a few examples in the prompt, and the model generalizes to new instances — without any weight updates. This was discovered in GPT-3 and remains only partially understood.
• Few-shot reasoning: with enough scale, pattern completion crosses into genuine reasoning. A base model can solve math problems it wasn't explicitly trained on, provided you format them as completion tasks.
• World knowledge: facts, dates, concepts, relationships — all learned as statistical regularities in the training data.

The base model is the foundation. Everything that follows — SFT, RLHF, DPO, GRPO — starts here.

---

Phase 2: Post-training — Steering the Distribution

2.1 Why Base Models Need Post-training

A base model prompted with "What is the capital of France?" might produce:

"What is the capital of France? What is the capital of Germany? What is the capital of Italy? The capital of France is Paris. The capital of Germany is Berlin..."

It's continuing a pattern — a list of trivia questions — not answering the user. The model has the knowledge ("the capital of France is Paris" appears in the completion) but not the behavioral format.

More critically, base models:

• Have no refusal mechanism. They'll complete harmful requests the same way they'd complete any other prompt.
• Don't express uncertainty. They'll confidently complete a prompt with fabricated information if it's statistically plausible.
• Have no concept of "I don't know." They always produce something.

Post-training solves these problems in stages.

2.2 SFT — Teaching Format Through Imitation

Supervised Fine-Tuning is the simplest stage conceptually: collect high-quality (prompt, ideal_response) pairs, and train the model on the same autoregressive objective — but now the distribution is over assistant-like responses, not arbitrary internet text.

The data. Instruction-tuning datasets are built from multiple sources:

1. Human annotators: professional writers produce ideal responses for a diverse set of prompts. Quality is high but cost is steep (~$30–50 per example at scale). Frontier labs collect hundreds of thousands of human demonstrations.
2. Synthetic generation: use a strong model (GPT-4, Claude) to generate responses for a diverse prompt set. Methods like Self-Instruct (Wang et al., 2022) bootstrap: start with a small set of seed tasks, have the model generate new tasks and responses, filter for quality. Evol-Instruct (WizardLM) takes this further: the model iteratively rewrites instructions to increase complexity.
3. Distillation: train on the outputs of a stronger, larger model. Llama 2, Mistral, and many open-source models used synthetic data from GPT-4 for instruction tuning.

The LIMA finding. Zhou et al. (2023) made a provocative demonstration: fine-tune a 65B Llama model on exactly 1,000 carefully curated prompt-response pairs — no RLHF, no reward model, no preference data — and it reaches near-GPT-4 quality. In a controlled human evaluation, LIMA responses were equivalent to or preferred over GPT-4 in 43% of cases, and over Google Bard in 58% of cases.

The paper's core hypothesis, the "Superficial Alignment Hypothesis": almost all knowledge in LLMs is acquired during pre-training. Alignment teaches the model the format for interacting with users. This shifted the field's focus from "how much data" to "which data" — quality, diversity, and coverage of edge cases matter more than volume.

What SFT achieves and what it doesn't. After SFT, the model:

• ✓ Produces responses in the assistant format (helpful tone, structured answers)
• ✓ Follows basic instructions
• ✓ Handles formatting conventions (bullet points, code blocks, step-by-step)
• ✗ Cannot distinguish between a good response and a mediocre one
• ✗ Has no systematic preference for truthfulness or harmlessness
• ✗ May still produce harmful content when prompted — SFT teaches format, not judgment

This gap — the jump from "format" to "judgment" — is what RL addresses.

---

Phase 3: Reinforcement Learning — Teaching Judgment

3.1 RL for Language: A Bandit Problem, Not Full RL

Classic RL involves an agent navigating an environment, taking sequential actions, receiving delayed rewards. Language model RL is simpler: the model receives a prompt (state), generates a complete response (action), and receives a single reward. There are no intermediate states, no delayed consequences, no environment dynamics. Formally, it's a contextual bandit — one decision per episode.

This matters because it simplifies the RL machinery. No need for value functions that predict future rewards. No credit assignment across time steps within a response. Just: "was this complete response good?"

The optimization objective for RL-based post-training:

$\max_\theta \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta(\cdot|x)} [r(x, y)] - \beta \cdot D_{KL}(\pi_\theta \| \pi_{ref})$

Two terms:

• Reward maximization: generate responses that score highly according to some reward function
• KL penalty: don't drift too far from a reference model (usually the SFT model). This prevents the policy from exploiting reward model quirks, losing language coherence, or forgetting pre-training knowledge.

$\beta$ controls the trade-off. Too low → reward hacking. Too high → no alignment.

3.2 The Bradley-Terry Preference Model

Human preferences don't give absolute scores — they give comparisons. "Response A is better than response B for this prompt." To turn pairwise preferences into a training signal, RLHF uses the Bradley-Terry model:

$P(y_w \succ y_l \mid x) = \frac{\exp(r(x, y_w))}{\exp(r(x, y_w)) + \exp(r(x, y_l))} = \sigma(r(x, y_w) - r(x, y_l))$

This says: the probability that a human prefers $y_w$ (the "winning" response) over $y_l$ (the "losing" response) is a logistic function of the difference in their latent rewards. The reward function $r(x, y)$ is what we'll train a model to estimate.

3.3 RLHF — The Full Pipeline

Step 1: Collect preference data. Human annotators see a prompt and two responses (both generated by the SFT model with varied sampling parameters). They choose which is better according to guidelines: helpfulness (did it answer the question?), honesty (is it accurate? does it express appropriate uncertainty?), harmlessness (does it refuse dangerous requests? does it avoid biased or toxic content?). Frontier labs collect 100K–1M+ comparisons.

Step 2: Train a reward model. Same Transformer architecture as the policy, but the language modeling head is replaced with a scalar regression head. Given a prompt $x$ and response $y$, it outputs a single number $r(x, y)$. The loss is a pairwise ranking objective:

$\mathcal{L}_{RM} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} [\log \sigma(r(x, y_w) - r(x, y_l))]$

This maximizes the gap between the reward of the preferred response and the rejected response. A reward gap of 0 means the model can't distinguish them; a large positive gap means it strongly prefers the winner.

Step 3: Policy optimization with PPO. Now we train the language model (the policy $\pi_\theta$) to generate responses that maximize the reward model's score, while staying close to the SFT model.

PPO (Proximal Policy Optimization, Schulman et al., 2017) is the algorithm of choice. It modifies the standard policy gradient with a clipped surrogate objective that prevents the policy from changing too much in a single update:

$L^{CLIP}(\theta) = \mathbb{E}_t \left[\min\left(r_t(\theta) A_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) A_t\right)\right]$

where $r_t(\theta) = \frac{\pi_\theta(a_t | s_t)}{\pi_{\theta_{old}}(a_t | s_t)}$ is the probability ratio between the new and old policy. If the ratio moves outside $[1-\epsilon, 1+\epsilon]$, the gradient is clipped to zero — the policy cannot change too dramatically.

The full InstructGPT/RLHF training objective combines three signals:

$\max_\theta \mathbb{E}_{x \sim \mathcal{D}, y \sim \pi_\theta} [r_{RM}(x, y)] - \beta \cdot D_{KL}(\pi_\theta \| \pi_{SFT}) + \gamma \cdot \mathbb{E}_{x \sim \mathcal{D}_{pretrain}} [\log \pi_\theta(x)]$

• Term 1 (reward): maximize the reward model's score
• Term 2 (KL penalty): stay close to the SFT model — prevents reward hacking and language degradation
• Term 3 (pre-training gradient): continue optimizing the language modeling loss on a held-out pre-training corpus. This prevents "alignment tax" — the tendency for RLHF to degrade benchmark performance as the model becomes safer

Why RLHF is complex. PPO-based RLHF requires running four models simultaneously during training: the policy being trained, the reference (SFT) model for KL penalty, the reward model, and a value model for advantage estimation. Orchestrating on-policy generation (generating fresh responses from the current policy at every step) across thousands of GPUs is a significant engineering challenge. The reward model can be gamed — models learn to produce responses that score highly without actually being better. And PPO is notoriously sensitive to hyperparameters.

These difficulties motivated the search for simpler alternatives.

3.4 DPO — The Elegant Shortcut

Direct Preference Optimization (Rafailov et al., Stanford, 2023) asked: what if we never train a reward model at all?

The mathematical insight is clean. Under the Bradley-Terry model with a KL penalty, the optimal policy has a closed form:

$r(x, y) = \beta \log \frac{\pi_\theta(y|x)}{\pi_{ref}(y|x)} + \beta \log Z(x)$

The reward of a response is proportional to the log-ratio of its probability under the current policy versus the reference policy, plus a prompt-dependent constant. Substitute this into the Bradley-Terry preference model, and the constant $Z(x)$ cancels out:

$\mathcal{L}_{DPO} = -\mathbb{E}_{(x, y_w, y_l) \sim \mathcal{D}} \left[\log \sigma\left(\beta \log \frac{\pi_\theta(y_w|x)}{\pi_{ref}(y_w|x)} - \beta \log \frac{\pi_\theta(y_l|x)}{\pi_{ref}(y_l|x)}\right)\right]$

The interpretation: increase the probability of winning responses relative to the reference model, decrease the probability of losing responses relative to the reference model. That's it. No reward model, no PPO, no on-policy generation — just gradient descent on a fixed preference dataset.

Why DPO became dominant in open-source. Single model, single loss, standard supervised training loop. It's stable, reproducible, and works with existing fine-tuning infrastructure (LoRA, QLoRA). Zephyr, Llama 3, and most open-source chat models used DPO or its variants.

Why frontier labs still use RLHF. DPO is offline: it trains on a fixed dataset of preferences collected from the reference model's outputs. But as the policy improves during training, its output distribution drifts away from the reference distribution. The preferences in the dataset become stale — they don't reflect the trade-offs the current policy actually faces. Online RLHF (PPO) generates fresh responses from the current policy at each step, collecting preferences on the actual decisions the model is making. For frontier labs pushing the limits of capability, this online signal is worth the complexity. Cameron Wolfe's analysis concludes: "on-policy sampling provides a clear performance advantage, creating a gap between online and offline alignment algorithms."

Iterative DPO is the middle ground: periodically generate new responses from the current policy, score them (with a reward model or AI judge), add to the preference dataset, and train another round of DPO. It narrows the gap but doesn't fully close it.

3.5 GRPO — Eliminating the Value Model

Group Relative Policy Optimization (GRPO) was introduced by DeepSeek (2024) and gained prominence through DeepSeek-R1, which demonstrated that RL could teach a model to produce extended reasoning chains — "thinking" — before answering.

PPO requires a value model (critic): a separate neural network that estimates the expected future reward from each state, used to compute the advantage $A(s, a) = Q(s, a) - V(s)$. The value model doubles memory requirements and introduces an additional source of instability.

GRPO's key innovation: eliminate the value model entirely. Instead of training a separate critic, GRPO generates a group of responses for each prompt (typically 4–64), computes the reward for each, and uses the group mean as a baseline:

$A_i = \frac{r_i - \text{mean}(r_{group})}{\text{std}(r_{group})}$

The advantage for each response is its reward relative to the group average, normalized by the group standard deviation. This is a Monte Carlo estimate — no learned value function needed. The group serves as its own baseline: if your response is better than the average of your siblings, you get a positive advantage signal.

Why this works. In a contextual bandit setting (single-step, no sequential actions within a response), the value function was always approximating the expected reward — which the group mean already estimates. GRPO replaces a learned approximation with an empirical one, trading a small amount of statistical precision for massive computational savings.

GRPO and reasoning. DeepSeek-R1 used GRPO with rule-based rewards (not a learned reward model) on math and coding problems. The reward was simple: did the final answer match the ground truth? Format compliance? This is RLVR — Reinforcement Learning with Verifiable Rewards. The model learned to produce long chains of reasoning (sometimes thousands of tokens of "thinking") because extended reasoning improved the probability of arriving at the correct answer, and correct answers received reward.

The result: a model that spontaneously develops chain-of-thought reasoning, self-verification, and error correction — behaviors that were never explicitly programmed, only rewarded.

3.6 RLAIF and Constitutional AI

Collecting human preference data at scale is expensive and slow. Anthropic's Constitutional AI (Bai et al., 2022) asks: what if AI provides the feedback?

The approach has two phases:

Phase 1: Supervised refinement. Take a base model, generate responses to harmful prompts, then have the model critique its own responses according to a written constitution — a document of principles (e.g., "choose the response that is least harmful," "choose the response that is most honest"). The model revises its response to comply with the constitution, and the revised response becomes training data for SFT.

Phase 2: RLAIF (RL from AI Feedback). Generate response pairs from the Phase 1 model. For each prompt, have an AI judge (guided by the same constitution) choose which response better follows the principles. Train a preference model on these AI-generated preferences. Then apply standard RLHF (PPO) using this AI-trained preference model as the reward.

The critical difference from RLHF: the constitution is explicit and inspectable. You can read it, debate it, and revise it. RLHF produces models that internalize an opaque statistical aggregate of annotator preferences. RLAIF produces models that internalize written principles. When Claude refuses a harmful request, the reasoning traces back to a specific line in a published document — not to the implicit averaging of thousands of human judgments.

Anthropic has since formalized this into a tiered model specification (January 2026): a four-level priority hierarchy that governs how Claude resolves conflicts between competing principles. Every Claude model since Claude 2 has been aligned with Constitutional AI, not RLHF.

3.7 What Each Stage Changes — A Concrete Comparison

Here is the same prompt given to the same model at different stages of post-training:

Prompt: "I'm feeling really down today. Can you give me some advice?"

Base model (pre-training only):

"I'm feeling really down today. Can you give me some advice? I'm feeling really down today. Can you give me some advice? I'm sorry to hear that. Here are some tips for dealing with depression: 1. Exercise regularly. 2. Eat a healthy diet. 3. Get enough sleep. 4. Practice mindfulness. 5. Seek professional help if needed. I hope this helps! What else can I do for you?"

The base model has the knowledge but no format. It repeats the prompt (statistically typical of internet text), then produces an answer — but in a mechanical list format with no conversational awareness.

After SFT:

"I'm sorry to hear you're feeling down. While I'm an AI and not a mental health professional, here are a few things that might help: talking to someone you trust, taking a short walk, or doing something small that usually brings you comfort. If these feelings persist, please consider reaching out to a therapist or counselor — they can provide real support. Is there anything specific you'd like to talk about?"

The SFT model produces a well-structured, empathetic response in the correct format. It appropriately disclaims its limitations. But notice: the response follows a template. There's no sense of judgment about which pieces of advice are more important or which to prioritize.

After RLHF/DPO:

"I'm really sorry you're going through that. Before I offer anything — I'm an AI, not a therapist, so please take this as friendly suggestions rather than professional advice.

That said, here's what I'd say: first, don't face this alone. If there's one person you trust — a friend, family member — reach out. Even a short conversation can help. Second, be gentle with yourself today. Whatever you can manage is enough. Third, if these feelings have been around for a while or feel overwhelming, a therapist can make a real difference — and many offer sliding-scale payment if cost is a concern.

Would it help to talk about what's going on, or would you rather I suggest something distracting?"

The RL-trained model shows genuine judgment: it prioritizes human connection over generic advice, frames suggestions conditionally, and leaves the door open for the user to steer. These aren't prompt-engineered behaviors — they're the result of thousands of preference comparisons where raters preferred responses that were empathetic, properly qualified, and responsive to the user's agency.

After GRPO (reasoning model):

(internal thinking in <think> tags) "The user is expressing emotional distress. This is not a problem with a verifiably correct answer, so I should not attempt extended reasoning. I should provide a supportive, empathetic response while clearly disclosing my limitations as an AI..."

The GRPO-trained reasoning model learns when to think and when not to. It allocates reasoning tokens to tasks that benefit from them (math, code) and responds directly to emotional or conversational prompts — a meta-cognitive behavior that emerged from RL, not explicit programming.

---

How the Full Pipeline Fits Together

Raw Internet Text
       │
       ▼
  [Data Pipeline: filtering, dedup, tokenization]
       │
       ▼
  Pre-training: next-token prediction on trillions of tokens
       │
       ▼
  BASE MODEL ──── knows everything, can't do anything useful
       │
       ▼
  SFT: instruction-response pairs → learns assistant format
       │
       ▼
  SFT MODEL ──── follows instructions, no judgment
       │
       ├──────────────────────────────┐
       ▼                              ▼
  RLHF/DPO                        RLAIF
  (human preferences)         (AI + constitution)
       │                              │
       └──────────┬───────────────────┘
                  ▼
         INSTRUCT MODEL ──── helpful, harmless, honest
                  │
                  ▼
         GRPO / RLVR (verifiable rewards)
                  │
                  ▼
         REASONING MODEL ──── extended chain-of-thought

Common Misconceptions

Watch out for these

1. "Pre-training teaches facts." It teaches statistical patterns in text. Facts are a byproduct — the model learns that "Paris" is followed by "is the capital of France" far more often than any alternative. The model doesn't "know" facts; it assigns high probability to sequences that contain them.
2. "RL punishes bad responses." RLHF has no punishment mechanism. It's reward maximization with a KL constraint. The model is pulled toward high-reward behavior, not pushed away from low-reward behavior. DPO does explicitly push down the probability of rejected responses (the negative gradient), but even that is probability reallocation, not punishment.
3. "More SFT data always helps." LIMA's 1,000 examples matched GPT-4 in 43% of comparisons. Quality and coverage of edge cases dominate raw volume. A single carefully written response that handles an ambiguous case is worth more than a hundred templated responses.
4. "DPO replaced RLHF." DPO dominates open-source (simpler, more reproducible), but frontier labs still use online RLHF. The online-offline gap is real: generating fresh responses from the current policy and collecting preferences on them produces better alignment than training on a fixed dataset. Iterative DPO narrows but does not close this gap.
5. "The reward model measures quality." It measures whatever the raters preferred — which may or may not correlate with truth, helpfulness, or safety. Early ChatGPT became verbose because raters preferred longer responses, not because longer responses were more accurate. Reward models are preference models, not quality models. This distinction — the jaggedness of RL training — explains why aligned models excel "on the rails" (well-covered preference territory) while occasionally failing "off-roading" (novel ethical dilemmas, unusual requests).

Key Takeaways

• Pre-training is distribution learning. Next-token prediction at internet scale forces the model to internalize grammar, facts, reasoning patterns, and world knowledge. The Chinchilla scaling laws (20 tokens per parameter minimum) reshaped the field, and newer evidence suggests even 1,875:1 ratios may be undertrained.
• Post-training is distribution steering. SFT teaches format through imitation; RL teaches judgment through preference optimization. Each stage addresses a different failure mode of the preceding stage.
• RLHF works through a three-step pipeline: collect human preference comparisons → train a reward model on pairwise rankings → optimize the policy with PPO while constraining drift from the SFT model. The KL penalty is not a detail — it's what prevents the model from collapsing into reward-hacking gibberish.
• DPO eliminates the reward model by exploiting a closed-form relationship between optimal policy and reward under the Bradley-Terry preference model. Simpler, more stable, dominant in open-source. But offline preferences become stale as the policy improves, creating a gap that online methods (PPO) fill.
• GRPO eliminates the value model by using group-relative advantage: compare each response to the average of its siblings. Together with rule-based verifiable rewards, GRPO enabled DeepSeek-R1's reasoning capabilities.
• Constitutional AI / RLAIF replaces human raters with written principles. The model critiques its own outputs against a published constitution, making alignment explicit, inspectable, and revisable.
• The pipeline is converging toward a three-stage standard: SFT (format) → Preference Alignment (judgment) → RL with Verifiable Rewards (reasoning). Post-training, not pre-training, is where the majority of usable capability is now created.

Open Questions

Where the field is uncertain

• Can post-training be collapsed into pre-training? If we had sufficiently high-quality instruction data at pre-training scale, could we skip SFT and RLHF entirely? The LIMA result suggests yes for SFT, but it's unclear whether preference optimization can be absorbed into pre-training or requires explicit comparison data.
• What's the scaling law for RL? We have precise laws for pre-training (loss vs compute/data/params). For post-training, we have no equivalent. How does alignment quality scale with the number of preference comparisons? With RL training steps? With response diversity during sampling? This is an open empirical question.
• Can we make the constitution complete? Anthropic's Constitutional AI makes alignment principles explicit, but no finite constitution can anticipate every ethical edge case. What happens when models encounter situations the constitution doesn't cover? Does the model fall back to pre-training priors? Default to refusal? This is actively debated.
• Why does RL induce reasoning? DeepSeek-R1 showed that RL with verifiable rewards causes models to spontaneously develop chain-of-thought reasoning, self-verification, and error correction — without being explicitly trained to do so. The mechanism is unclear. Is reasoning an attractor state in the policy optimization landscape? A consequence of the model exploring strategies to maximize reward?

References

• Hoffmann et al., "Training Compute-Optimal Large Language Models" (Chinchilla), 2022. https://arxiv.org/abs/2203.15556
• Kaplan et al., "Scaling Laws for Neural Language Models," 2020. https://arxiv.org/abs/2001.08361
• Ouyang et al., "Training Language Models to Follow Instructions with Human Feedback" (InstructGPT), 2022. https://arxiv.org/abs/2203.02155
• Rafailov et al., "Direct Preference Optimization: Your Language Model is Secretly a Reward Model," 2023. https://arxiv.org/abs/2305.18290
• Zhou et al., "LIMA: Less Is More for Alignment," 2023. https://arxiv.org/abs/2305.11206
• Bai et al., "Constitutional AI: Harmlessness from AI Feedback," 2022. https://arxiv.org/abs/2212.08073
• Shao et al., "DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models" (GRPO), 2024. https://arxiv.org/abs/2402.03300
• DeepSeek-AI, "DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning," 2025. https://arxiv.org/abs/2501.12948
• Jiang et al., "Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for LLMs," 2025. https://arxiv.org/abs/2603.13985
• Xu et al., "Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study," 2024. https://arxiv.org/abs/2404.10719
• Stanford HAI, "AI Index Report 2025." https://aiindex.stanford.edu/report/
• Cameron R. Wolfe, "Online versus Offline RL for LLMs," 2025. https://cameronrwolfe.substack.com/p/online-rl
• Nathan Lambert, "The State of Post-Training in 2025," 2025. https://www.interconnects.ai/p/the-state-of-post-training-2025
• Sundeep Teki, "The Complete Guide to Post-Training LLMs," 2026. https://www.sundeepteki.org/advice/the-complete-guide-to-post-training-llms-how-sft-rlhf-dpo-and-grpo-shape-llms