How we fine-tuned a faster model for Patterns on Baseten
Source
datadoghq.com
Date
Why it matters
A narrow, high-volume LLM step cost about 80% of p99 cost. Fine-tuning a 9B model on traced production spans lifted judge pass rate from 45% to 93% and cut latency from about 45s to under 1s.
Key takeaways · AI-distilled
Benchmarking first showed the base Qwen 3.5 9B could already summarize: 78 of its 83 failures ran out of tokenThe chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.Full definition → or could not be parsed. fine-tuningTaking a trained model and training it a bit more on your own examples so it gets better at one specific job.Full definition → taught it to answer directly, and every fine-tuned answer finished within the token limit.
Datadog built the training set from Patterns runs with several different clustering instructions so the model would learn summarization rather than one instruction, and it generalized across instructions in evaluation.
To prevent leakage, pup finetune assigns training, validation and test splits by trace, so all rows of one trace and repeated copies of a request land in the same split. A per-trace row cap keeps busy traces from dominating.
Before scoring, Datadog's LLMA large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.Full definition → judge must pass a check on up to four test cases: accept the recorded answer and reject two deliberately corrupted versions of it.
The coding AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → driving pup finetune shows a four-line preview of what runs, who receives the data, the cost and the exact command, and waits for approval before every paid step. It treats trace content as data so instructions inside traces cannot steer it.
Terms in this piece · Glossary
LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.