Migrate a password reset flow to modern libraries
As a point of reference, I tried an experiment with Claude Code and the latest Opus the other day. Task: As a point of reference, I tried an experiment with Claude Code and the latest Opus the other day. It was work for my company, so this was using API tokens and not the individual user plans that have non-commercial terms.A simple task, migrating a typical password reset flow as part of updating a long-lived web application from legacy libraries and software architecture to modern equivalents, apparently cost roughly the same as 2 months of Pro subscription, over the equivalent of about half a working day in wall time.It produced code of decent quality at a small scale, but it wasn’t always on point architecturally. It also had a tendency to drift off topic and try to tangle up other changes it decided should be made with the main change we were supposed to be working towards. So even for a routine task, based on a plan developed using the harness first and with the agents working under close supervision, a near-SOTA model is still producing quality on par with a decent mid-level developer but substandard for anyone senior+ in this case.Moreover, based on a direct comparison with other migration tasks of similar complexity that I’d already done by hand, it was actually a bit slower overall to work this way. I had to babysit Claude throughout and review everything it proposed carefully, both to avoid subtle errors (it would have made several) and to prevent drifting off track. I also had to spend a significant amount of time cleaning up its final output to an acceptable standard after the session. Those two overheads more than cancelled out the much faster code generation an LLM offers under favourable conditions.So for now, I remain sceptical about these high multiples of improved productivity that I keep seeing claimed online from people who are apparently writing almost everything using AIs now. I could certainly have achieved a multiple of my normal productivity by YOLOing everything without reviewing it in detail and then accepting the output code without tidying anything up. However, I doubt this codebase would still have been good enough for normal human developers to work on it reasonably after even 10 or 20 AI-led sessions like that. The architecture would have degraded significantly and the test suite would have been large and largely pointless. And again, this wasn’t rocket science in this experiment, it was completely unremarkable maintenance of a relatively small and simple web application.; tools: Claude Code. Human role: The original post does not say how a person stayed involved.; limitations: High operational costs and significant time spent on babysitting the AI negate productivity gains for complex or architectural tasks. The model requires strict human oversight to avoid subtle errors and scope creep, making it unsuitable for autonomous execution in production environments. Source reliability is based on a single anecdotal experiment, so results may vary by codebase complexity and developer expertise..
Tools
Published on Agent Work Atlas:
Overview
As a point of reference, I tried an experiment with Claude Code and the latest Opus the other day.
Workflow
As a point of reference, I tried an experiment with Claude Code and the latest Opus the other day. It was work for my company, so this was using API tokens and not the individual user plans that have non-commercial terms.A simple task, migrating a typical password reset flow as part of updating a long-lived web application from legacy libraries and software architecture to modern equivalents, apparently cost roughly the same as 2 months of Pro subscription, over the equivalent of about half a working day in wall time.It produced code of decent quality at a small scale, but it wasn’t always on point architecturally. It also had a tendency to drift off topic and try to tangle up other changes it decided should be made with the main change we were supposed to be working towards. So even for a routine task, based on a plan developed using the harness first and with the agents working under close supervision, a near-SOTA model is still producing quality on par with a decent mid-level developer but substandard for anyone senior+ in this case.Moreover, based on a direct comparison with other migration tasks of similar complexity that I’d already done by hand, it was actually a bit slower overall to work this way. I had to babysit Claude throughout and review everything it proposed carefully, both to avoid subtle errors (it would have made several) and to prevent drifting off track. I also had to spend a significant amount of time cleaning up its final output to an acceptable standard after the session. Those two overheads more than cancelled out the much faster code generation an LLM offers under favourable conditions.So for now, I remain sceptical about these high multiples of improved productivity that I keep seeing claimed online from people who are apparently writing almost everything using AIs now. I could certainly have achieved a multiple of my normal productivity by YOLOing everything without reviewing it in detail and then accepting the output code without tidying anything up. However, I doubt this codebase would still have been good enough for normal human developers to work on it reasonably after even 10 or 20 AI-led sessions like that. The architecture would have degraded significantly and the test suite would have been large and largely pointless. And again, this wasn’t rocket science in this experiment, it was completely unremarkable maintenance of a relatively small and simple web application.
Reported result
The task took approximately half a working day and cost roughly the equivalent of two months of a Pro subscription. The author found the process slower than manual migration due to the overhead of constant supervision and post-session cleanup. Code quality was deemed comparable to a mid-level developer but substandard for senior roles, with the AI frequently drifting off-topic or introducing unnecessary changes.
Human role
The original post does not say how a person stayed involved.
Limitations
High operational costs and significant time spent on babysitting the AI negate productivity gains for complex or architectural tasks. The model requires strict human oversight to avoid subtle errors and scope creep, making it unsuitable for autonomous execution in production environments. Source reliability is based on a single anecdotal experiment, so results may vary by codebase complexity and developer expertise.
Try it
I need to migrate a [specific feature, e.g., password reset] in my [language/framework] application from [legacy library] to [modern library]. Use Claude Code with the latest Opus model via API. First, generate a detailed migration plan and wait for my approval. Then, execute the code changes step-by-step. After each step, pause and show me the diff and your reasoning. I will review for architectural correctness and scope drift. Do not proceed to the next step until I confirm the current one is clean. If you encounter errors, stop and ask for guidance rather than attempting complex self-corrections. Finally, provide a summary of changes and any manual cleanup I need to perform. Ensure no unrelated files are modified.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.