Factory AI

Droid now runs a STRIDE-based security review on every non-draft PR, posting inline findings with severity, CWE references, and fixes; Factory cites disclosed real-world findings like CVE-2026-42876 as evidence it works.

Legacy-Bench is a benchmark developed by Factory.ai to evaluate how well frontier AI coding agents can understand, maintain, and modify legacy software systems written in languages like COBOL, Fortran, and Java 7. It targets the kind of mission-critical infrastructure — financial settlement systems, telecom routing, and similar — that still runs on decades-old code, testing whether AI agents can handle real-world legacy engineering challenges rather than modern greenfield codebases.

Factory built Signals, which uses LLM judges to detect friction and delight in agent sessions (like a user rephrasing a request five times) without exposing raw conversation text to human reviewers, then feeds detected friction back so the agent fixes its own behavior.

Factory Analytics gives engineering leaders dashboards on Droid usage: token spend by model, tool-call patterns (Read/Edit/Execute), adoption stickiness, and output like files edited and PRs merged, built on OpenTelemetry for export to existing stacks.

Factory's Legacy-Bench, now part of Fireworks' Specialized Intelligence Index, scores frontier models on debugging and migrating COBOL, Java 7, BASIC, C89, Fortran, and Assembly, showing wide gaps and a model that silently miscalculated a payroll deduction while passing most tests.

Factory describes the organizational model behind its enterprise Droid deployments: a commercial enterprise account sitting above nested organizations that each scope their own integrations, model access, residency, and admins to mirror a company's real structure.

Factory's Agent Effectiveness connects coding-agent sessions to Jira/Linear/GitHub delivery data, showing whether cycle times are dropping, whether agent effort matches planned priorities, and which issues and PRs each session produced.

Factory launched Factory Router, which automatically picks the cheapest capable model per agent task and fails over across providers, reporting 99% of Claude Opus 4.7's Terminal-Bench 2 pass rate at 20% lower cost, and 96% of its Legacy-Bench pass rate at 25% lower cost.

Factory describes its shift from single coding agents to 'software factories,' introducing Missions for long-horizon multi-agent workflows and Factory Desktop for full local-machine access, built around model-independent routing and self-improving feedback loops across the SDLC.

Factory's Legacy-Bench tests coding agents on COBOL, Java 7, Fortran, BASIC, C89, and Assembly tasks from real enterprise domains. Frontier agents score 16.9-42.5% here versus 70%+ on SWE-bench Verified, doing worst on languages like COBOL where bugs don't throw visible errors.

Factory opened Droid Computers, persistent cloud or bring-your-own machines that keep an agent's filesystem, credentials, and running services intact between sessions, betting against the ephemeral-sandbox model most cloud coding agents use.

Factory launched Connectors, managed no-config access to apps like Slack, Sentry, Salesforce, Notion, and Figma, alongside Personas, which route each role (engineering, product, finance) to relevant Connectors during onboarding, working alongside existing MCP integrations.

Factory's Automated QA skill drives an app like a real user (filling forms, hitting endpoints, checking session persistence) and posts one evidence-backed pass/fail report as a PR comment, catching flows CI misses.

Factory details how Droid avoids re-summarizing entire conversations as they grow: it anchors persistent summaries to specific messages and only compresses the newly-dropped span each time, keeping compression cost roughly constant instead of scaling with conversation length.

Factory launched Factory Private and a pending FedRAMP option, letting enterprises run its Droid coding agents' control plane and data inside their own VPC, on-prem, or air-gapped network instead of only Factory's hosted cloud.

Factory adds User Groups to its Organization Model. Groups sync from an identity provider and carry usage limits and model access across organizations, resolved by individual override, group, then org default. Manageable through the admin console and Organization API.

Factory built a probe-based evaluation to test whether context compression actually preserves what an agent needs, asking recall, artifact, continuation, and decision questions after compression, and found structured summarization beat comparable approaches from OpenAI and Anthropic.

Factory argues model routing belongs inside the agent harness, not the API gateway, since the harness has session history and cache state. Production data: routing cut aggregate cost 58% and median latency from 81s to 49s with no drop in task success across eight quality metrics.

Factory benchmarked 13 LLMs as PR reviewers across 50 real PRs from five open-source projects: GPT-5.2 and Opus 4.6 lead on F1, but cheaper models like Kimi K2.5 and MiniMax M2.7 hit 75-86% of that quality at a fraction of the cost.

Factory explains the rationale behind Missions: agents are highly reactive to their append-only context, so broad tasks accumulate irrelevant context, and an agent that implemented something is a biased judge of its own work, fixed via narrowly scoped workers plus independent validators.

Factory makes Automations generally available: describe a recurring workflow, pick a schedule, Slack, GitHub or webhook trigger, and Droid runs it with a chosen model and machine. Templates cover ticket-to-PR and CI triage.

Factory introduces Agent Effectiveness, a view of how Factory usage maps to delivery speed, work priorities and shipped artifacts across an engineering organization.

An article from Factory.ai arguing that AI agents can provide significant leverage for software development when used correctly, and outlining an approach to building software with agents.

Factory had its Droid agent analyze 780,000 of its own web search and fetch calls, finding three-quarters of queries are software-development related, with documentation and learning material dominating overall usage.