Fine-tuning experiments on CoT controllability

Low chain-of-thought controllability is a safety assumption for monitoring reasoning. A small fine-tune raising it suggests the assumption may not be robust to accidental optimization pressure.

• chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
• fine-tuning — Taking a trained model and training it a bit more on your own examples so it gets better at one specific job.

• articleWhen Chain-of-Thought Helps and When It Hurts: An Empirical Investigation of the Serial-Depth Bottleneck in LLM ReasoningTughanbulut Kurtulush
• articleWe tested every major AI reasoning system. There is no clear winner.Mike Knoop
• videoControlling powerful AIAnthropic

Checking sign-in…

Loading comments…