Crossing the Threshold: How OpenAI and Anthropic Approach Autonomy and Safety in Financial Agents
Source
latent.space
Author
Richard MacManus
Date
Why it matters
Argues that safe agent autonomy depends on where controls and human review are placed, not on how many restrictions are imposed, which informs how you design agent guardrailsThe checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.Full definition →.
Key takeaways · AI-distilled
OpenAI's Joseph Kim described a 'threshold of usefulness' where a model is reliable enough to save time, and predicted that in 2027 finance will rely less on Excel and PowerPoint and agents will handle investment idea generation and trading.
Two Sigma's John Feminella warned that approval fatigue both slows teams and adds risk, and described composable controls including auto-mode classifiers that clear routine calls and escalate only consequential ones, with deterministic rules for clear cases.
Anthropic's Keon Shahab framed an AI agentAn AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.Full definition → trust layer in three parts: identity and permissions, a sandboxAn isolated environment where AI-generated code or agent actions run without being able to touch anything real.Full definition → approval layer before the agent acts, and checks that it still works as variables change. His trade-break agent proposes order amendments but does not make them.
Bloomberg's Andrey Rybka cited an agent that treated text in a marketplace review as an instruction to stock a product, and argued for deterministic controls outside the model: scoped credentials, tool-call checks, egress limits, audit trails, budgets and kill switches.
Terms in this piece · Glossary
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
guardrails — The checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.
sandbox — An isolated environment where AI-generated code or agent actions run without being able to touch anything real.
