LLM distillation demystified: a complete guide

Distillation works best when the student has a specific job. A banking assistant may route requests into a fixed intent set. A coding model may work inside one repository and toolchain. A support agent may follow a defined tool policy. Each job gives the team concrete examples, edge cases, and success criteria for training and evaluation.

During LLM distillation, a teacher model produces the data used to train a smaller student. That data may contain labels, responses, worked solutions, preferences, token probabilities, or internal representations. The student trains on those examples and is evaluated on a separate test set drawn from the intended use case.

A successful student preserves the teacher’s accuracy on the chosen task while reducing latency, serving cost, or hardware requirements. Teacher quality, example coverage, student capacity, and held-out evaluation determine the result.

LLM distillation basics

Multi-billion parameter language models pre-trained on millions of documents have changed the world. Users can ask ChatGPT or Grok any number of questions and often get useful answers.

LLMs’ flexibility dazzles, but most AI problems don’t require flexibility. They require accuracy, speed, and efficiency.  LLMs, while amazing, tend to be slow and expensive. That’s where distillation comes in.

What is LLM distillation?

Knowledge distillation trains a student model on examples or model signals produced by a teacher. The teacher expresses its behavior through outputs, probability distributions, or internal representations. The training objective teaches the student to reproduce the patterns represented in that data.

The available signal determines the method:

Each signal fits a different training problem. Intent classification may require only a final label. Multi-step coding may benefit from worked solutions or tool traces. Rationales add value when they expose decisions you can verify; unchecked rationales can also add errors and shortcuts. Choose the teacher signal by working backward from the task and its evaluation.

Why would you use LLM distillation?

LLMs like GPT-4, Gemini, and Llama demonstrate incredible power, but also suffer notable drawbacks:

• Cost. Multi-billion parameter LLMs are expensive to host, and even more expensive to access via API.
• Speed. Due to the quantity of calculations necessary, full-sized LLMs can be slow.
• Infrastructure headaches. Hosting private versions of the largest available LLMs means wrangling and coordinating significant resources.

By distilling an LLM, data science teams can build derivative models that are easier to host, cheaper to run, and much more responsive.

Limitations of LLM Distillation

Training-set coverage determines where the student works reliably. Familiar prompts may look strong while rare intents, ambiguous requests, adversarial inputs, and longer reasoning paths remain weak. Near-duplicate examples add volume while leaving those gaps untouched.

Teacher mistakes become training examples unless the pipeline filters them. Domain experts define valid outcomes, identify high-risk cases, and review fluent answers that contain technical errors. The teacher scales candidate generation; experts decide which outputs are suitable for the deployed task.

Student capacity sets another limit. A small model may learn one focused capability well and lose accuracy when a training run combines several unrelated capabilities. Compare the student with the teacher and a directly fine-tuned baseline on held-out task quality, class- or difficulty-level failures, latency, and end-to-end serving cost.

Permission depends on the teacher provider, model license, and intended deployment. Terms may treat training on model outputs, commercial use, redistribution, and derivative models differently. Review the current terms for each teacher and student before generating the dataset.

Practical LLM distillation for classification challenges

Basic distillation rarely yields production-grade models. In a Snorkel case study classifying user intents for a banking chatbot, our engineers started with labels from Google’s PaLM 2 to achieve an F1 of 50 as a baseline. That’s an impressive performance for an out-of-the-box model—especially considering the case study called for 77 classes—but it would not meet any bank’s bar for deployment.

With a little prompt engineering (encouraging the LLM to behave as an expert in banking and giving one example per label), the team boosted the PaLM 2’s F1 score to 69. That’s much closer to production-grade performance, but not close enough. What, then, can a data scientist do with this not-quite-there dataset? Enrich it.

How to enrich training data with targeted human labeling

A model that achieves a 69 F1-score learned decision boundaries from broadly accurate data.  That same model can help spot questionable training examples.

By turning the model back on its training data, data scientists can find where the model predicts its own training data incorrectly and where it makes predictions with low confidence. They can then isolate these points for human review and update.

While few people enjoy the tedium of manually labeling data, this approach can drastically reduce the number of individual examples humans must inspect. Data science teams that want to further reduce the need for human labeling can employ supervised or semi-supervised learning methods to relabel likely-incorrect records based on the patterns set by the high-confidence data points.

In cases where the high-confidence subset of data appears sufficiently robust, data scientists could cast aside all records tagged as likely-incorrect and train their model with the remainder.

Enriching data sets with multi-signal distillation

Data scientists can distill labels from multiple sources of signal. At the simplest level, they can send the same documents and well-engineered prompts to several publicly available LLMs. The best of the breed might achieve 69 F1 on its own, but several LLMs “voting” on the right answer typically achieve better performance than any individual model.

Data scientists can also target sections of the data set with additional, narrowly focused prompts or heuristic rules. For example, our engineer In the banking chatbot case added a rule that applied the label “card_arrival” to utterances that contained the words “card” and “track.” They also could have approached this challenge with a narrow prompt sent to an LLM, such as asking “Does this text ask about the arrival of an ATM card?”

While this approach is potent (we use it in our Snorkel Flow AI data development platform), it requires data scientists to reconcile conflicting signals. Simple voting yields diminishing returns as data scientists add more sources of signal. Snorkel Flow uses an advanced version of an approach called weak supervision.

Generative LLM distillation with prompts and responses

Generative LLM distillation with prompts and responses works similarly to LLM distillation for classification models. The main difference between the two practices is that data scientists extract responses from the teacher model instead of labels.

Once a data scientist gathers responses from the teacher model, fine-tuning the student model works the same way it would with human-generated prompts and responses.

Teacher choice carries a licensing question at this step. Provider terms differ on training with model outputs, commercial deployment, and derivative models, and those terms change. Confirm the current terms for your specific teacher and intended deployment before generating a response dataset.

A different approach: knowledge distillation

Knowledge distillation uses a larger “teacher” neural network model to sharpen the performance of a smaller “student’ neural network.

The concept of knowledge distillation for neural networks stretches back to a 2015 paper, and made a serious mark on data science well before the arrival of ChatGPT. In 2019, researchers introduced DistillBERT, which used knowledge distillation during pre-training “to reduce the size of a BERT model by 40%, while retaining 97% of its language understanding capabilities.”

At the time of this writing, DistillBERT was among the 20 most downloaded models on Hugging Face.

How does knowledge distillation work?

Knowledge distillation focuses on training the student model’s probability distribution to mimic those of the teacher model. This differs from the approaches discussed above, which care only about the teacher LLM’s output. Some knowledge distillation approaches train the student model directly on the “soft target” of the teacher model’s probability distribution. Others use the teacher model’s textual output and convert it to numerical vectors. 

This has had significant impacts on non-generative models through achievements like DistillBERT, but can have downsides when used to shrink to generative models; the smaller model can overfit to the teacher model’s training examples, resulting in inaccurate or repetitive responses.

Better knowledge distillation for generative models

While still an active area of study, researchers have developed a promising method for improving knowledge distillation for generative models with an approach called “MiniLLM.”

Like some previous knowledge distillation approaches, MiniLLM uses “soft targets” extracted directly from the teacher model instead of its textual output. However,  previous “soft target” knowledge distillation methods encouraged the student model to mimic the entire probability distribution. MiniLLM encourages the student to focus only on high-probability outcomes.

In experiments, MiniLLM led to improvements of as much as 15 points over previous training methods and sometimes produced student models that outperformed their teachers.

LLMs distilling themselves: context distillation

Researchers and data scientists have achieved significant performance gains in language generation by cleverly engineering prompts. They may include a message that encourages the prompt to respond like a particular avatar (“you are an expert data scientist…”), to include example prompts and responses, or to use a “scratch pad” to allow the LLM to work out some logic before rendering a final answer.

However, the model does not learn from these highly-engineered prompts. The performance improvements dissipate as soon as the model responds.

Researchers at the University of California at Berkeley found a way to change this, which they call “context distillation.” They built heavily-engineered prompts that ended with simple questions, such as “add these numbers.” Then, they stripped the prompt of its engineering and reduced the response to only its final answer to create a new data set that they used to fine-tune the model.

When they then asked the fine-tuned models the same questions with no added context, they found that the rate at which the model answered correctly increased.

In their research, the “teacher” and “student” models were the same size. However, this approach theoretically extends to distilling the responses to context-rich prompts generated by larger models to bolster smaller models.

Generative LLMs for predictive tasks: distilling step-by-step

While distillation has proven again and again to be useful in bootstrapping smaller models that perform nearly as well as teacher models at a tiny fraction of the inference cost, the approach comes with one minor limitation: it requires a large amount of raw data.

While a lack of labeled data bottlenecks many AI projects, distillation can be bottlenecked by a lack of unlabeled data. Imagine, for example, that you want a model to classify contract clauses into a dozen categories, but you have very few raw examples to train from. In a strict LLM to classifier model use case, you likely couldn’t get the performance you need.

However, the distilling step-by-step approach developed by researchers at Google and Snorkel AI allows data scientists to fine-tune a small generative model for classification tasks on as little as one-eighth as much data as traditional fine-tuning would require.

The approach works by asking the teacher model to return not only its answer but also the rationale behind its answer. The training pipeline then directs the student model to do the same—to yield both a final response and reasoning for that response. The pipeline updates the model’s weights according to both portions of its output.

While powerful, this approach only works when distilling to a smaller generative model. It does not allow data scientists to mirror the classification accuracy of something as powerful as GPT-4 in a footprint as small as DistillBERT. However, researchers were able to use a model up to 700 times smaller than their teacher model (PaLM, at 540 billion parameters)  while achieving comparable results.

LLM Distillation in 2026

Reasoning models have expanded distillation datasets beyond final answers. Teams can retain worked solutions, critiques, and tool traces when intermediate decisions matter to the task. Every retained step needs verification because a correct answer can still contain a faulty inference, leaked clue, or shortcut.

The DeepSeek-R1 technical report describes smaller dense models trained on reasoning samples generated by the larger model. Its math and coding results support a specific conclusion: response-based distillation can transfer substantial capability on the evaluations represented in the study. Broader claims require additional tasks, models, and test conditions.

Distilling Step-by-Step demonstrates a different use of teacher reasoning. The teacher supplies both a label and a natural-language rationale, giving the student two supervised targets. On the classification tasks studied, the additional signal improved data efficiency relative to standard fine-tuning. In practice, rationale quality becomes a data-development problem. Generated explanations need filtering, deliberate coverage of difficult cases, and evaluation outside the teacher-generated set.

Compute economics can change the decision. Distillation adds teacher-inference and student-training costs. Distillation Scaling Laws find more favorable economics when a capable teacher already exists or several students reuse its outputs. A team training one teacher for one student should compare that plan with direct training at the same quality threshold.

Diagram Brief

The signal available from the teacher determines how the student can be trained. Output access supports generated targets; logits and internal representations require white-box access.

Frequently Asked Questions About LLM Distillation

Learn More

Follow Snorkel AI on LinkedIn, Twitter, and YouTube to be the first to see new posts and videos!