• Do the prompt engineering first, but for a reason people miss: it stress-tests your eval system. If prompting gets you there and your evals confirm it, stop, because you now have both an answer and a harness you trust.
• A domain-specific eval harness is the prerequisite, not the follow-up. Without one you cannot tell whether a fine-tune helped, so teams write fine-tuning off when what they actually lack is measurement.
• Fine-tuning is not an open-weights-only move. Perplexity.AI and CaseText both fine-tuned GPT-3.5 through the API.

• fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
• eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
• agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Gives a clear decision framework for whether to invest in fine-tuning: build a domain-specific eval harness first, exhaust prompt engineering (partly to validate your evals), and recognize that fine-tuning skeptics often work on general-purpose products where it wouldn't help anyway.

• articleDo Automated Evals Work?
• articleWhy I Stopped Using nbdev
• articleEvals Skills for Coding Agents
• articleYour AI Product Needs Evals

• postStart evals dirty: five examples beat waiting for the perfect harnessAndrew Ng
• videoYour Fine-Tuned Model Is Tech Debt: A 50x ROI House of Cards — Dan Bjornn, Lease EndAI Engineer
• articleThe Goldilocks Of Pragmatic Understanding Fine Tuning Strategy Matters For Implicature Resolution By Llms 2022 10 26Cohere

Checking sign-in…

Loading comments…