Large language model training: three phases that shape LLM training

Large language model training uses several kinds of data. Pretraining teaches patterns through token prediction across large collections of text and code. Instruction tuning gives the model demonstrations of useful responses. Post-training supplies preferences or verifiable rewards that shape which responses the model selects.

Pretraining builds broad language and domain capability. Instruction tuning teaches the model to respond to requests and follow constraints. Post-training uses preferences or verifiable rewards to reinforce chosen behaviors. Each phase addresses a different class of error.

Use the failed behavior to choose the phase. Missing knowledge points to pretraining or targeted supervised data. Poor instruction following points to demonstrations. Reward hacking or preference regressions point to the post-training data, reward, or verifier. Held-out evaluation shows whether the change improved the intended behavior.

I recently gave a talk at Snorkel AI’s second Enterprise LLM Summit about the problems that can surface when the data for these three labels is not properly aligned. The talk included a short explainer of the three training phases, which the Snorkel team excerpted as a separate video. I’ve summarized the main points below.

Phase 1: self-supervised learning for language understanding

Self-supervised learning, the first stage of training, is what traditionally comes to mind when we talk about language modeling.

It involves feeding the model with vast amounts of unannotated or raw data and having it predict ‘missing’ pieces of it. The model learns something about language and the domain of the data to generate plausible answers.

For example, if we feed the model with text from a weather website and ask it to predict the next word, it has to understand something about the language and the weather domain. In the presentation, I gave the example sentence “A flash flood watch will be in effect all _____.”

In an intermediate state, the model ranks a list of predictions, from likely answers (“day”, “night”, “hour”) to less plausible ones (“month”), and even nonsensical ones (“giraffe”) that should be assigned very low probability. This process is called self-supervision (as opposed to unsupervised learning) because there is a specific right answer—the word that actually appeared in the text we collected—which was “night” in my example. Self-supervision is similar to unsupervised learning in that it can use abundant, unannotated data.

Phase 2: Instruction tuning

Instruction tuning trains the base model on demonstrations: an instruction or context paired with a useful response. People, existing supervised tasks, model generation, or a combination of sources can supply the examples. The demonstration format defines this phase.

Task composition controls what the model practices. A mixture dominated by short factual questions teaches concise responses and provides little practice with long instructions containing several constraints. Scaling Instruction-Finetuned Language Models showed that diverse tasks, templates, and input formats can improve generalization to unseen tasks. The mixture needs examples of the capabilities expected at deployment.

Experts resolve demonstrations with several plausible answers. They specify mandatory constraints, acceptable alternatives, and consequential mistakes. Model generation then expands the dataset around those decisions.

The same demonstrations can also train a smaller specialist through LLM distillation. Instruction tuning adapts a model to a task format and response style. Distillation uses teacher-produced data to transfer selected behavior into a student model.

Phase 3: Post-training

An instruction-tuned model can produce fluent answers and still choose the wrong behavior. Post-training, often discussed as LLM alignment, introduces feedback that distinguishes stronger outputs from weaker ones. The feedback may come from human comparisons, model-assisted critiques, written rules, executable tests, or changes in an environment.

In the RLHF pipeline described in Training Language Models to Follow Instructions With Human Feedback, people compare candidate responses, those comparisons train a reward model, and reinforcement learning updates the language model to produce responses with higher predicted reward. This approach can represent qualities that are easier to compare than to specify exactly, but the learned reward inherits the ambiguities and blind spots of the preference data.

Direct Preference Optimization learns from preferred and rejected responses and trains the policy directly against those pairs. This removes the separate reward-model and online reinforcement-learning stages used in the conventional RLHF pipeline. DPO belongs in post-training and uses a different optimization procedure from RLHF.

Verifiable tasks support another feedback path. Code can run against tests, mathematical answers can be checked against ground truth, and an agent’s work can be scored against environment state. DeepSeekMath introduced GRPO as a policy-optimization method that compares groups of responses generated for the same prompt. GRPO defines the policy update. Human ratings, learned judges, and programmatic verifiers can each supply the reward.

Each feedback source creates its own failure modes. Preference data can be inconsistent. Reward models can be exploited. Deterministic graders can award a shortcut that passes the check while missing the intended task. Researchers define the task, test the reward against adversarial cases, and evaluate the updated model outside the training loop.

Where RLVR fits

RLHF is useful when the target quality depends on human judgment, such as tone, helpfulness, or whether one response is preferable to another. Reinforcement learning with verifiable rewards (RLVR) applies when success can be checked programmatically: a mathematical answer matches ground truth, code passes tests, a structured output satisfies a schema, or an agent leaves an environment in the required state.

People define the RLVR training system. Researchers choose the task mix, specify acceptable outcomes, build the verifier, and test whether a passing result represents the intended behavior. A weak verifier can reward a shortcut just as weak preference data can train the wrong behavior.

Snorkel’s Learning from Less study examines how dataset size, diversity, and complexity affect RLVR in low-data and compute-constrained settings. The study reports up to fivefold sample efficiency for mixed-complexity training sets compared with easier-only sets. Mixed complexity helped in the reported low-data settings because the tasks remained solvable often enough to produce useful reward variation. Apply the finding under the model, budget, and task family used in the training run.

Three phases. Three techniques. One improved model.

Pretraining, instruction tuning, and post-training teach through different data and objectives. Diagnose the missing behavior, choose the phase that can teach it, and define the evidence that will show improvement.

Evaluation provides the return path. A held-out test may expose a missing capability, a brittle refusal, a verifier exploit, or a regression in an adjacent domain. Researchers can then collect expert-authored examples, generate targeted variants, revise the task mixture, or harden the grader before another training run. Data quality remains a continuing research problem across the training stack.

The learning signal changes across LLM training. Held-out evaluation identifies the next data problem and sends researchers back to the phase that can address it.

Frequently Asked Questions About Large Language Model Training

How Are Large Language Models Trained?

Large language models first learn statistical patterns by predicting tokens across large collections of text, code, or other sequences. Many models are then instruction-tuned on demonstrations and post-trained with preferences, critiques, rules, or verifiable rewards.

Pretraining builds broad capability, instruction tuning teaches task-following behavior, and post-training shapes which responses the model selects.

What Is the Difference Between Pretraining, Fine-Tuning, and Post-Training?

Pretraining builds a base model through large-scale prediction objectives. Fine-tuning updates that model on a targeted dataset; instruction tuning is one form of fine-tuning. Post-training shapes behavior after pretraining and may include supervised tuning, preference optimization, or reinforcement learning. Teams use the labels differently, so name the objective and data whenever precision matters.

Is DPO the Same as RLHF?

DPO and RLHF can both learn from preference pairs. Conventional RLHF fits a reward model from comparisons and uses reinforcement learning to optimize the policy against that reward. DPO trains the policy directly with a preference objective and removes the separate reward-model and online reinforcement-learning stages.

What Is GRPO?

Group relative policy optimization is a reinforcement-learning method that compares several responses generated for the same prompt and updates the model based on their relative rewards. It is often used with verifiable rewards in reasoning tasks, but the concepts are separate: GRPO is an optimization method, while verifiability describes how a reward is checked.

How Is RLVR Different From RLHF?

RLHF learns a reward signal from human demonstrations or preference judgments. RLVR assigns rewards from outcomes that can be checked programmatically, such as a correct answer, passing code, valid structured output, or a confirmed change in environment state.

Use human feedback when quality depends on judgment, interpretation, or preference. Use verifiable rewards when success can be specified and checked reliably. A training system can combine both: verifiable rewards for measurable task completion and human feedback for qualities that resist a deterministic rule.

Where Does Synthetic Data Fit in LLM Training?

Synthetic describes how an example was produced. Generated documents or code can extend a pretraining corpus. Generated demonstrations and worked solutions can support instruction tuning. Generated candidates, critiques, and trajectories can be ranked or verified during post-training.

Its value depends on coverage, filtering, and independent evaluation. A larger generated dataset can amplify the generator’s errors just as easily as it can expand useful supervision.

Learn More

Follow Snorkel AI on LinkedIn, Twitter, and YouTube to be the first to see new posts and videos!