Vibeleaderboard
← All Intel
Intel / article

How we fine-tuned a faster model for Patterns on Baseten

Source
datadoghq.com
Date
Why it matters

A narrow, high-volume LLM step cost about 80% of p99 cost. Fine-tuning a 9B model on traced production spans lifted judge pass rate from 45% to 93% and cut latency from about 45s to under 1s.

Key takeaways · AI-distilled
  • Benchmarking first showed the base Qwen 3.5 9B could already summarize: 78 of its 83 failures ran out of or could not be parsed. taught it to answer directly, and every fine-tuned answer finished within the token limit.
  • Datadog built the training set from Patterns runs with several different clustering instructions so the model would learn summarization rather than one instruction, and it generalized across instructions in evaluation.
  • To prevent leakage, pup finetune assigns training, validation and test splits by trace, so all rows of one trace and repeated copies of a request land in the same split. A per-trace row cap keeps busy traces from dominating.
  • Before scoring, Datadog's judge must pass a check on up to four test cases: accept the recorded answer and reject two deliberately corrupted versions of it.
  • The coding driving pup finetune shows a four-line preview of what runs, who receives the data, the cost and the exact command, and waits for approval before every paid step. It treats trace content as data so instructions inside traces cannot steer it.
Terms in this piece · Glossary
  • LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
  • AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
  • fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
  • token — The chunk of text a model reads and writes in — roughly three-quarters of a word — and the unit AI usage is billed in.
Read the source www.datadoghq.com
Recommended reads
Comments

Checking sign-in…

Loading comments…