Warp introduced Factories in closed beta on August 18, then announced Factory Benchmarks on September 3. The useful question is which workflow produces acceptable changes at a sustainable cost on your repository.
What the factory organizes
The launch announcement describes versioned factory definitions, agents for triage, specification, implementation and review, and integrations with development tools. Hosting, inference and data retention are configurable choices; bring-your-own infrastructure and zero-data-retention arrangements should not be mistaken for identical defaults on every account.
The official walkthrough below makes the product's task flow and configuration more concrete than a static illustration. It is a vendor demonstration, not evidence that its agents correctly handle every change.
Introducing Warp Factories: official walkthrough
Official presentation from Warp. The YouTube player loads only when you request it.
Watch on YouTube · Maintainer’s announcement. Loading the player connects to YouTube, which applies its own privacy policy.
Compare configurations on the same work
The September 3 benchmarks announcement describes tasks, configurations, scorers and repeated runs, including reconstruction from past work and its original repository state. Product access remains early, and supported harness combinations should be checked in the actual beta rather than inferred from the platform's longer-term ambitions.
An original trial design could use ten fixed tasks, two configurations and three repetitions: 60 runs. Start each run from the same task-specific commit and provide the same required information. Keep a separate held-out set for checking whether changes to prompts or skills generalize beyond the tasks used to tune them.
The cover depicts that proposed test matrix. It contains no measured Warp results. Repetitions matter because one successful response cannot show how reliably the same configuration handles a task.
Count failed work as well as accepted changes
Suppose configuration A costs $60 across its attempts and produces eight accepted changes; B costs $45 and produces five. Their costs per accepted change are $7.50 and $9 respectively. B is cheaper in total but more expensive by that particular outcome measure. These fictional totals exclude human review and infrastructure, which should be added for a full cost comparison.
Acceptance should include deterministic checks appropriate to the change, targeted regression coverage and review of the actual diff. An LLM judge can help apply a rubric, but its score does not prove correctness. Record retries and rejected work, and review changes to the scoring rubric separately from changes to the agent being scored. That avoids rewarding a factory merely for making its own evaluation easier.
Add September 3 Factory Benchmarks, distinguish optional hosting/ZDR and vendor internal claims; explain repeated held-out evaluation and embed official walkthrough.