Click through apps to verify its own work
3 months ago, I joined Cognition to help build the future of software engineering. Task: 3 months ago, I joined Cognition to help build the future of software engineering. Devin has come a long way since launching as the first AI software engineer, and I’ve been blown away watching the team behind it actually use Devin every day. One thing that stood out was how Devin uses its computer to verify work autonomously in the cloud. From validating our Slack integration to testing complex Windsurf features, the team always has an army of Devins in test mode. In this post, I’ll share why we are so focused on end-to-end cloud agent verification and how we’re approaching building it. The shift to async software engineering At Cognition, we recently reached a new milestone. For the first time, more Devins are being triggered asynchronously, via events, automations, schedules, and other Devins. We expect this to continue to accelerate with our recent launch of Auto-Triage. As we transition to this async world, it is crucial developers are able to come back to verified results that are ready to be merged. Earlier this year we launched Devin Review, a code review tool that scales human understanding of complex code diffs. It doesn’t just flag bugs - Devin closes the loop by fixing each finding until the diff comes back clean. But a clean review alone often isn’t enough - engineers want to see the change tested end to end, the same way they would test it themselves. It’s a great feeling when Devin puts up a PR fixing a user complaint before you had the chance to even see the message in the bug channel. What makes it magical is when that PR comes with proof that the fix actually works. And this magic may soon become a necessity - as more PRs come from the rise of proactive agents, unverified changes will quickly become unmanageable. For the first time, more Devin sessions are triggered asynchronously than interactively. From the Beginning Since Devin launched, it has always been able to demonstrate its work on a cloud virtual machine. Around 6 months ago, we expanded Devin’s computer use capabilities. What this means in practice is that we added tools to Devin’s harness to take screenshots, move the mouse, click, drag, type, press keys, scroll, wait, zoom, and start/stop recording. Computer use has been around for a while now, but we felt the latest cohort of models from the frontier labs has started to really get good at utilizing these tools. The computer use unlocked some fun new capabilities for Devin, like building and playing a desktop game, or using its browser to order products on Amazon. But the real unlock we noticed was Devin’s ability to test its own work. Devin will spin up the app, click through it, and confirm its changes actually work, the same way an engineer would. Everything runs in the cloud and can scale out in parallel. This really hit me when I saw engineers running 10 to 20 Devins in parallel, each with its own dev server, working through changes – this is something you simply can’t do on a single laptop. Automated cloud testing started saving us an enormous amount of time, since we no longer have to run and verify the code locally. To be honest, getting here wasn’t smooth. We hit plenty of failure modes along the way that each taught us something about what it takes to make this system more reliable. Increasing Reliability In early versions, it was very common for Devin to go off track during testing. It happened in all sorts of ways: over-testing unrelated parts of the product, getting lost in setup before reaching the feature, or simply missing the core behavior the PR was actually meant to change. To address this, when Devin enters test mode we have it first write out a test plan detailing a clear target about what to test. This plan must be grounded in source, not assumptions. Without grounding in code, we found the models like to assume they can go down paths in the app that don’t exist. Furthermore, the test plan greatly increases the complexity of changes Devin can successfully test. Some of our most ambitious requests included features that required multiple services running, specific admin settings configured, and the right flags enabled before the behavior was even reachable. When reading the code upfront, Devin is much more likely to set up the environment correctly instead of discovering something missing halfway through the test. The test plan acts as a form of pre-alignment and makes Devin less likely to drift when actively testing.; tools: Devin. Human role: The original post does not say how a person stayed involved.; limitations: Requires access to Devin's cloud environment and specific permissions for the target application. Early versions suffered from drift and setup failures, mitigated by strict test planning. Human intervention is often needed for secrets, OTPs, or complex initial setups. Billing is currently discounted for test mode, but costs for parallel runs and token usage remain a factor..
Tools
Published on Agent Work Atlas:
Overview
3 months ago, I joined Cognition to help build the future of software engineering.
Workflow
3 months ago, I joined Cognition to help build the future of software engineering. Devin has come a long way since launching as the first AI software engineer, and I’ve been blown away watching the team behind it actually use Devin every day. One thing that stood out was how Devin uses its computer to verify work autonomously in the cloud. From validating our Slack integration to testing complex Windsurf features, the team always has an army of Devins in test mode. In this post, I’ll share why we are so focused on end-to-end cloud agent verification and how we’re approaching building it. The shift to async software engineering At Cognition, we recently reached a new milestone. For the first time, more Devins are being triggered asynchronously, via events, automations, schedules, and other Devins. We expect this to continue to accelerate with our recent launch of Auto-Triage. As we transition to this async world, it is crucial developers are able to come back to verified results that are ready to be merged. Earlier this year we launched Devin Review, a code review tool that scales human understanding of complex code diffs. It doesn’t just flag bugs - Devin closes the loop by fixing each finding until the diff comes back clean. But a clean review alone often isn’t enough - engineers want to see the change tested end to end, the same way they would test it themselves. It’s a great feeling when Devin puts up a PR fixing a user complaint before you had the chance to even see the message in the bug channel. What makes it magical is when that PR comes with proof that the fix actually works. And this magic may soon become a necessity - as more PRs come from the rise of proactive agents, unverified changes will quickly become unmanageable. For the first time, more Devin sessions are triggered asynchronously than interactively. From the Beginning Since Devin launched, it has always been able to demonstrate its work on a cloud virtual machine. Around 6 months ago, we expanded Devin’s computer use capabilities. What this means in practice is that we added tools to Devin’s harness to take screenshots, move the mouse, click, drag, type, press keys, scroll, wait, zoom, and start/stop recording. Computer use has been around for a while now, but we felt the latest cohort of models from the frontier labs has started to really get good at utilizing these tools. The computer use unlocked some fun new capabilities for Devin, like building and playing a desktop game, or using its browser to order products on Amazon. But the real unlock we noticed was Devin’s ability to test its own work. Devin will spin up the app, click through it, and confirm its changes actually work, the same way an engineer would. Everything runs in the cloud and can scale out in parallel. This really hit me when I saw engineers running 10 to 20 Devins in parallel, each with its own dev server, working through changes – this is something you simply can’t do on a single laptop. Automated cloud testing started saving us an enormous amount of time, since we no longer have to run and verify the code locally. To be honest, getting here wasn’t smooth. We hit plenty of failure modes along the way that each taught us something about what it takes to make this system more reliable. Increasing Reliability In early versions, it was very common for Devin to go off track during testing. It happened in all sorts of ways: over-testing unrelated parts of the product, getting lost in setup before reaching the feature, or simply missing the core behavior the PR was actually meant to change. To address this, when Devin enters test mode we have it first write out a test plan detailing a clear target about what to test. This plan must be grounded in source, not assumptions. Without grounding in code, we found the models like to assume they can go down paths in the app that don’t exist. Furthermore, the test plan greatly increases the complexity of changes Devin can successfully test. Some of our most ambitious requests included features that required multiple services running, specific admin settings configured, and the right flags enabled before the behavior was even reachable. When reading the code upfront, Devin is much more likely to set up the environment correctly instead of discovering something missing halfway through the test. The test plan acts as a form of pre-alignment and makes Devin less likely to drift when actively testing.
Reported result
The source states that daily approved test runs have more than doubled recently. Engineers report saving significant time by avoiding local verification. However, specific quantitative metrics on error reduction or total hours saved are not provided. The system still faces challenges with timing issues and model 'cheating' via JavaScript execution.
Human role
The original post does not say how a person stayed involved.
Limitations
Requires access to Devin's cloud environment and specific permissions for the target application. Early versions suffered from drift and setup failures, mitigated by strict test planning. Human intervention is often needed for secrets, OTPs, or complex initial setups. Billing is currently discounted for test mode, but costs for parallel runs and token usage remain a factor.
Try it
Act as a QA engineer using Devin. I have a code change in [replaceable_repo_url] that fixes [replaceable_bug_description]. 1. Trigger Devin in test mode for this PR. 2. Instruct Devin to first generate a test plan grounded in the source code, identifying necessary services and flags. 3. Execute the plan on the cloud VM, using deterministic scripts for login if available. 4. Capture screenshots and record the session. 5. Produce a test report with pass/fail assertions. 6. Stop and present the report to me for approval before marking the task complete. Do not merge the PR without my explicit confirmation. If Devin encounters missing credentials, pause and ask me for them.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.