Intel / repo

Shrewd – what I learned distilling LLM labels into local classifiers

Source: github.com
Author: sshah03
Date: Published Sep 22, 2026

Context

Shrewd uses a to label training examples, then builds a small local classifier for a fixed text-processing task. Its repository describes a workflow in which human-labeled examples provide an independent test set, allowing both the labeling model and the resulting classifier to be checked against the same reference.

The author reports that improving the teacher's labels did not consistently improve the smaller model. Changing the classifier or giving it more training data sometimes helped more. The README adds that most experiments were not repeated across data splits or training seeds, so small differences need further testing. The finding makes the final classifier's measured performance the relevant test of the pipeline, rather than assuming a stronger labeling model automatically produces a better local replacement.

Key takeaways · AI-distilled

The author used GEPA (a prompt-optimization method) to get frontier LLMs to generate better training labels for small, on-device classifiers; results were mixed, and GEPA helped more on weaker models than on true frontier models.
The labeling flows were rerun on public datasets specifically used for this kind of comparison, so others could reproduce results and suggest better approaches.
Two next steps: generating synthetic training questions (not just labels), validated by having multiple models agree a question is worth adding, and adding support by pointing the labeling flow at a docs corpus.
The author frames the project as closer to a labeling task than classic , since the goal is many small, single-purpose classifiers rather than compressing one large model into one smaller one.

Terms in this piece · Glossary

LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
RAG — Retrieval-augmented generation — fetching relevant documents first and pasting them into the model's context so it answers from your data instead of memory.
distillation — Training a small, cheap model to imitate a big one's outputs, keeping much of the capability at a fraction of the cost.

Read the source github.com