Update a docs page of AI CLI tools with Devin
My first impressions review of Devin: The user experience, how good it is at completing tasks, and how it compares against a human engineer. Task: My first impressions review of Devin: The user experience, how good it is at completing tasks, and how it compares against a human engineer. Devin just became publicly available, where anyone can just sign up and pay to get access without going through a sales call. Here’s my first impressions of Devin. This post consists of 3 parts: the user experience, the performance, and the pricing. It will take about 15 minutes to read. User Experience - Superb This is hands-down the best user experience I’ve had for an AI coding solution. Onboarding The first obvious thing you notice that it is extremely well designed for software engineers as users. This is very apparent from the onboarding step: The Devin interface guides you through a series of steps (with video guide as well) to setup the repository. How to install dependencies, how to run lint and tests, what are the things to take note for the repository. Here are some minor UX issues (no default value provided on how to install dependencies, I had to type that), but overall it is very clearly thought-out and as a developer you understand why it is done this way, and how the configurations you setup during onboarding relates to VM provisioning, snapshots, etc. Repo notes During repo setup, Devin automatically generates a repo note as part of its knowledge base on what it should know about the repository. You can edit it manually, and when working on tasks and interacting with users, Devin will continuously give you suggestions to update the knowledge base so that your instructions and rules are followed across sessions. It is like a memory component for Devin. Separately to the repo note in knowledge base, Devin also has an automatically generated index of the repository to understand on a high level the code organization, structure and what each component does in the repository. I suspect this is useful when doing Retrieval-augmented generation (RAG) on the codebase to figure out which are the files that are relevant to the task at hand. New Session Interface To start a new task, you are greeted with a huge chat window with suggestions at the bottom. This is really good when you want to describe the requirements in details. I find this user experience superior to other AI coding apps that are based on VS Code, which puts code at the center of the UI. When using Devin, I can focus more on describing what I want, instead of worry about the code. You can also give Devin tasks directly on Slack, and chat with Devin there. But I will focus on the experience of using the Devin web app in this post, as it is more interesting. Main Interface Once Devin starts working on the task, you get to the main interface. The main interface for Devin is very well designed from an engineer’s perspective. On the left you have a chat session, and on the right you have Devin’s Workspace. You can collapse either one of them to make the other one full screen. Two-way Realtime Chat On the left panel, you can interact with Devin via two-way communication: You can at anytime tell Devin to change something (use a different approach while Devin is working on the task), or update your requirements. You can at anytime tell Devin to change something (use a different approach while Devin is working on the task), or update your requirements. Devin will proactively seek out clarifications (when something is not right) and ask for your permission to do something (storing sensitive information). Devin will proactively seek out clarifications (when something is not right) and ask for your permission to do something (storing sensitive information). The cool thing is that Devin will respond in realtime to your messages and update its current plan to accommodate new information you have provided. It’s like you can interrupt it at anytime (hopefully Devin won’t get annoyed by my micro-management). Devin’s Workspace On the right panel, you have the Devin’s Workspace. On the first tab, you can “follow Devin’s actions” by looking at what Devin is doing. The rest of the tabs are very developer-centric, and allows you to “monitor” Devin’s actions. Broadly speaking, there are a few tools that Devin can utilize and take actions on: Shell - Executing a shell commandI am glad that Devin supports multiple shells, this allowed it to run the server while executing curl against it to test the logic.Devin is proficient with various git commands, gh commands (to interact with GitHub PRs and CI), and bash commands.; tools: Devin. Human role: The original post does not say how a person stayed involved.; limitations: Prerequisites include a public Devin subscription ($500/month for 250 ACUs) and a compatible code repository. Constraints include Devin's inability to access web pages requiring human-triggered authentication and potential performance degradation in long sessions. The underlying LLM may fail to use retrieved knowledge even when available. Human approval is required for sensitive information handling and final code commits..
Tools
Published on Agent Work Atlas:
Overview
My first impressions review of Devin: The user experience, how good it is at completing tasks, and how it compares against a human engineer.
Workflow
My first impressions review of Devin: The user experience, how good it is at completing tasks, and how it compares against a human engineer. Devin just became publicly available, where anyone can just sign up and pay to get access without going through a sales call. Here’s my first impressions of Devin. This post consists of 3 parts: the user experience, the performance, and the pricing. It will take about 15 minutes to read. User Experience - Superb This is hands-down the best user experience I’ve had for an AI coding solution. Onboarding The first obvious thing you notice that it is extremely well designed for software engineers as users. This is very apparent from the onboarding step: The Devin interface guides you through a series of steps (with video guide as well) to setup the repository. How to install dependencies, how to run lint and tests, what are the things to take note for the repository. Here are some minor UX issues (no default value provided on how to install dependencies, I had to type that), but overall it is very clearly thought-out and as a developer you understand why it is done this way, and how the configurations you setup during onboarding relates to VM provisioning, snapshots, etc. Repo notes During repo setup, Devin automatically generates a repo note as part of its knowledge base on what it should know about the repository. You can edit it manually, and when working on tasks and interacting with users, Devin will continuously give you suggestions to update the knowledge base so that your instructions and rules are followed across sessions. It is like a memory component for Devin. Separately to the repo note in knowledge base, Devin also has an automatically generated index of the repository to understand on a high level the code organization, structure and what each component does in the repository. I suspect this is useful when doing Retrieval-augmented generation (RAG) on the codebase to figure out which are the files that are relevant to the task at hand. New Session Interface To start a new task, you are greeted with a huge chat window with suggestions at the bottom. This is really good when you want to describe the requirements in details. I find this user experience superior to other AI coding apps that are based on VS Code, which puts code at the center of the UI. When using Devin, I can focus more on describing what I want, instead of worry about the code. You can also give Devin tasks directly on Slack, and chat with Devin there. But I will focus on the experience of using the Devin web app in this post, as it is more interesting. Main Interface Once Devin starts working on the task, you get to the main interface. The main interface for Devin is very well designed from an engineer’s perspective. On the left you have a chat session, and on the right you have Devin’s Workspace. You can collapse either one of them to make the other one full screen. Two-way Realtime Chat On the left panel, you can interact with Devin via two-way communication: You can at anytime tell Devin to change something (use a different approach while Devin is working on the task), or update your requirements. You can at anytime tell Devin to change something (use a different approach while Devin is working on the task), or update your requirements. Devin will proactively seek out clarifications (when something is not right) and ask for your permission to do something (storing sensitive information). Devin will proactively seek out clarifications (when something is not right) and ask for your permission to do something (storing sensitive information). The cool thing is that Devin will respond in realtime to your messages and update its current plan to accommodate new information you have provided. It’s like you can interrupt it at anytime (hopefully Devin won’t get annoyed by my micro-management). Devin’s Workspace On the right panel, you have the Devin’s Workspace. On the first tab, you can “follow Devin’s actions” by looking at what Devin is doing. The rest of the tabs are very developer-centric, and allows you to “monitor” Devin’s actions. Broadly speaking, there are a few tools that Devin can utilize and take actions on: Shell - Executing a shell commandI am glad that Devin supports multiple shells, this allowed it to run the server while executing curl against it to test the logic.Devin is proficient with various git commands, gh commands (to interact with GitHub PRs and CI), and bash commands.
Reported result
The source does not provide measured results for this specific documentation update workflow. While the author reports Devin completed simple to medium tasks, they noted issues with low recall in identifying all files needing updates and occasional infinite editing loops. No specific metrics on time saved or error rates for docs updates are provided. A measurable acceptance check would be to track the number of manual corrections required per documentation update task.
Human role
The original post does not say how a person stayed involved.
Limitations
Prerequisites include a public Devin subscription ($500/month for 250 ACUs) and a compatible code repository. Constraints include Devin's inability to access web pages requiring human-triggered authentication and potential performance degradation in long sessions. The underlying LLM may fail to use retrieved knowledge even when available. Human approval is required for sensitive information handling and final code commits.
Try it
I want to use Devin to update a specific documentation page in my [replaceable: repository URL] for [replaceable: project name]. First, guide me through the onboarding process to set up dependency installation and linting commands. Then, assign a task to update [replaceable: specific docs file path] with new information about [replaceable: feature or tool]. Monitor Devin's workspace actions in real-time. If Devin misses files or makes errors, provide immediate feedback in the chat. Before finalizing, review the proposed changes locally. Do not commit or push any changes until I explicitly approve the diff. Start with a dry run on a single file to verify the workflow.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.