Intel / article

Is Fine-Tuning Still Valuable?

Source: hamel.dev
Author: Hamel Husain
Date: Published Mar 27, 2024

Context

is most useful when the problem is a recurring pattern of behavior the model needs to learn. In a March 2024 account, Hamel Husain points to specialized syntax, style and output rules. Supplying changing facts through retrieval addresses a different need: giving the model relevant information when it answers.

The prerequisite is an set specific to the product. Prompt experiments help test whether that evaluation can detect improvements before a team invests in training. If prompting already meets the measured requirements, it may be sufficient; persistent failures on specialized rules provide a clearer reason to investigate fine-tuning.

Key takeaways · AI-distilled

Do the prompt engineering first, but for a reason people miss: it stress-tests your eval system. If prompting gets you there and your evals confirm it, stop, because you now have both an answer and a you trust.
A domain-specific eval harness is the prerequisite, not the follow-up. Without one you cannot tell whether a fine-tune helped, so teams write fine-tuning off when what they actually lack is measurement.
Fine-tuning is not an open-weights-only move. Perplexity.AI and CaseText both fine-tuned GPT-3.5 through the API.

Terms in this piece · Glossary

fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.
eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
agent harness — The scaffolding around a model that turns it into a working agent — the loop, the tools it can call, and the rules for when to stop.

Key quotes

“Generally speaking, fine-tuning works best to learn syntax, style and rules whereas techniques like RAG work best to supply the model with context or up-to-date facts.”
Hamel Husain

“It’s impossible to fine-tune effectively without an eval system which can lead to writing off fine-tuning if you haven’t completed this prerequisite.”
Hamel Husain

“The reason for doing lots of prompt engineering is that it’s a great way to stress test your eval system!”
Hamel Husain

“I’m a big believer in using the simplest approach to solving a problem. I just don’t think you should write off fine-tuning yet.”
Hamel Husain

Read the source hamel.dev