Explains how attestation-gated keys can keep user prompts and model weights unreadable outside verified inference and API servers. Useful for engineers assessing data-privacy guarantees of hosted LLMs.
Key takeaways · AI-distilled
User data must be readable at two points: the API Server, which tokenizes prompts and runs most request logic, and the inferenceRunning a trained model to get answers — the phase where AI is actually used, as opposed to trained.Full definition → Server on accelerators. Model weights are sensitive only on the Inference Server.
Because not all accelerators fully support confidential computing, Anthropic is exploring a small trusted model loader in a separate hypervisor-isolated VM that alone decrypts data, leaving the rest of the inference server untrusted.
The loader presents itself as a 'virtual accelerator' and only runs programs signed by Anthropic's secure CI server, which ensures the code was reviewed by multiple engineers.
The report's trusted environment includes hardware-isolated encrypted memory and disabled debugging features; Anthropic stresses the work is early and too soon to forecast as specific product features.
Terms in this piece · Glossary
inference — Running a trained model to get answers — the phase where AI is actually used, as opposed to trained.
