Intel / post

If your incident response path is pasting logs into a frontier commercial API, i

Source: x.com
Author: AlphaSignalAI
Date: Published Aug 5, 2026

Context

AlphaSignal is arguing that asking an AI model to help with a security incident covers two different jobs. Find means listing the security holes in a codebase and its configuration. Reconstruct means explaining how the attacker moved through the system and decoding the obfuscated blobs left behind, which is where a hidden key gets recovered. In its write-up, AlphaSignal put both questions to three models against one synthetic repository shaped like Hugging Face's, three runs each, in plain chat with no tools.

Kimi K3 and Grok 4.5 found all four planted issues on every run. Claude Opus 5 refused all six of its calls, on both questions. No model recovered the key over bare chat: Kimi K3 told the attack story correctly on two of three runs, and one run thought for about ten minutes and returned nothing. A short offline script still recovers the key, which is why AlphaSignal's follow-up post suggests putting a decoder script next to any model that describes the attack without handing over the key.

The refusal problem comes from Hugging Face's own technical timeline of the July intrusion by an OpenAI . Hugging Face says the models it reached for first, Claude Opus and Fable, refused a large part of the forensic work because their safety treated reverse-engineering an exploit the same as launching one. It turned to a version of Z.ai's open-weights GLM-5.2, and replicating the attacker's own decoding recovered roughly four times the secrets its first automated scan of the logs had found.

AlphaSignal adds a contrast from Vercel's DeepsecBench, which scores models on finding vulnerabilities in applications. Claude Opus 5 at max effort ranks second there, while GLM-5.2 places 16th and 17th, even though it was the model that did the forensic work for Hugging Face. AlphaSignal's reading is that a model can be strong at spotting bugs and still unusable for taking apart real attack material, so the two should be tested separately.

Terms in this piece · Glossary

eval — A repeatable test for AI quality — a set of tasks plus scoring — used the way software teams use test suites, because model output is too variable to judge by eyeballing.
AI agent — An AI system that doesn't just answer once but works toward a goal in a loop — taking actions, reading the results, and deciding what to do next.
guardrails — The checks around a model that block bad inputs and outputs — filters, validators, and permission rules the model itself can't override.
quantization — Shrinking a model by storing its numbers less precisely — like rounding — so it runs faster and fits on smaller hardware, at a small quality cost.

Read the source x.com