Try Devin on research tasks (mostly failed)
Hamel Husain. Task: The Answer.AI team tested Devin, an autonomous AI software engineer, on 20 real-world tasks including data integration, research, and code modification. They initiated tasks via Slack or a web interface, monitoring Devin's asynchronous execution in a Docker environment. Human involvement was limited to providing credentials, reviewing outputs, and deciding when to abandon failed attempts. The workflow involved assigning specific coding or research goals and waiting for Devin to produce code or documentation.; tools: Devin. Human role: The original post does not say how a person stayed involved.; limitations: Devin struggles with tasks requiring deep context understanding of existing codebases or internal tooling. It fails to identify fundamental blockers, leading to wasted time on impossible requests. The tool is asynchronous, making real-time debugging difficult. Human approval is critical before accepting any code or deployment. Source reliability is high as it is a first-hand account from a technical team, but results may vary by task complexity..
Tools
Published on Agent Work Atlas:
Overview
Hamel Husain
Workflow
The Answer.AI team tested Devin, an autonomous AI software engineer, on 20 real-world tasks including data integration, research, and code modification. They initiated tasks via Slack or a web interface, monitoring Devin's asynchronous execution in a Docker environment. Human involvement was limited to providing credentials, reviewing outputs, and deciding when to abandon failed attempts. The workflow involved assigning specific coding or research goals and waiting for Devin to produce code or documentation.
Reported result
The team reported 14 failures, 3 successes, and 3 inconclusive results out of 20 tasks. Early successes included pulling Notion data to Google Sheets and creating a planet tracker. However, scaling up revealed significant issues: Devin often produced overly complex, unusable code, hallucinated non-existent features (e.g., Railway deployment capabilities), and spent days on impossible tasks without recognizing limitations. The authors concluded that Devin lacked a reliable niche for their workflow.
Human role
The original post does not say how a person stayed involved.
Limitations
Devin struggles with tasks requiring deep context understanding of existing codebases or internal tooling. It fails to identify fundamental blockers, leading to wasted time on impossible requests. The tool is asynchronous, making real-time debugging difficult. Human approval is critical before accepting any code or deployment. Source reliability is high as it is a first-hand account from a technical team, but results may vary by task complexity.
Try it
I want to test Devin on a small, well-defined task to evaluate its reliability. My business context is [replaceable: e.g., migrating data from Notion to Google Sheets]. The source files or URLs are [replaceable: e.g., Notion API docs, Google Cloud Console access]. The target users are [replaceable: e.g., internal data analysts]. The output location is [replaceable: e.g., a new Google Sheet]. Please follow these steps: 1. Assign a single, small task to Devin via Slack or the web interface. 2. Monitor the process in real-time. If Devin asks for credentials, provide them securely. 3. If Devin encounters a limitation or seems stuck for more than [replaceable: e.g., 1 hour], stop the task and reassess. 4. Review the output code or data. Do not deploy or publish without human verification. 5. Document the time taken and any errors encountered. Measure quality by checking if the output is functional and simple. Success is defined as completing the task without hallucinating features or requiring excessive human intervention to fix errors. If inputs are missing, ask me for clarification before starting. Do not proceed with complex or ambiguous tasks.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.