The Intellectual History Behind Chain-of-Thought Monitoring
- Source
- x.com
- Date
New SAIL Blog post: CoT Monitoring: Where Does a Hot Safety Problem Come From? @peterbhase and @ChrisGPotts trace the history of a big idea in AI Safety
@peterbhase @ChrisGPotts The post: https://t.co/Bn7ASHZ9Ap
Context
monitoring means using an automated system, often a prompted , to read another model's written-out reasoning and flag it as potentially unsafe. Stanford SAIL's post links to a longer write-up by Hase and Potts tracing how that idea became a prominent AI safety strategy.
The authors flag as unusual that a 41-author agenda-setting paper on the topic appeared only a few months after the first arXiv mention of 'CoT monitoring,' since large multi-author position papers on an idea more typically follow years of separate work converging. Their own , stated as an interpretation rather than a documented fact, is that informal conversation among researchers, concentrated in a few specific hubs, spread the idea faster than the usual slow accumulation of published results would explain.
- chain-of-thought — Having a model write out intermediate reasoning steps before its answer, which markedly improves performance on hard problems.
- LLM — A large language model — the neural network behind tools like Claude and ChatGPT, trained on huge amounts of text to predict what comes next.
- inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
articleThe First Few Tokens Matter: Understanding Chain-of-Thought ControllabilityHugh Van Deventer, Dmitrii Troitskii, Laura Ruis- postNew SAIL Blog post: R&B-EnCoRe: What Should a Robot Actually Think About…Stanford AI Lab
articleWe tested every major AI reasoning system. There is no clear winner.Mike Knoop
Checking sign-in…
Loading comments…