Fill forms and review tax filings with Codex computer use
OpenAI engineers explain how Codex's computer-use feature clicks, browses and fills forms across your desktop, and why it took years to get reliable. Task: OpenAI engineers explain how Codex's computer-use feature clicks, browses and fills forms across your desktop, and why it took years to get reliable. What is computer use in OpenAI’s Codex? Computer use is the capability that lets an AI agent operate a computer the way a person does: moving a cursor, clicking buttons, scrolling pages, and filling in forms inside real applications and websites, rather than calling an API behind the scenes. In OpenAI’s Codex, this started as a developer-facing feature built on coding agents that ran visible scripts. As Codex expanded into a full desktop app for a broader audience, the same underlying capability got a friendlier face: a pointer that visibly clicks around the screen instead of exposing the raw automation script. TL;DR Computer use began as a niche developer tool where Codex showed the actual scripts and selectors it used to click through an app, which worked but wasn’t approachable for non-technical users. The interface changed as the audience grew, replacing visible code with a simple on-screen pointer so people could watch the agent work without needing to understand what’s happening underneath. Reliability crossed a tipping point in the last year or so, moving computer use from what one OpenAI engineer called an “impressive party trick” that was slow to something fast enough to use casually and constantly. Computer use now acts as a universal connector for tasks that have no API or MCP integration, like government paperwork, PDFs, or legacy web forms that will likely never get a dedicated connector. Direct integrations (Slack, email, and similar connectors) still matter because they’re often faster and more token-efficient than clicking through a UI, so OpenAI is investing in both approaches rather than picking one. Internal adoption at OpenAI happened unevenly by function, with coding teams switching first because their work lived in local command-line tools, and other teams like legal following once the model got good enough at reasoning over large amounts of material. The bigger shift is behavioral, according to OpenAI staff: the habit that matters most is catching yourself before doing a task manually and routing it to the agent instead. How did computer use evolve from a party trick to a reliable tool? According to OpenAI engineers who build the Codex desktop app, computer use has existed in some form for a while, but it stayed a developer tool for a long time because it needed technical users who wanted to see the scripts, selectors, and clicks the model was running under the hood. That transparency worked for engineers debugging automation, but it doesn’t hold up when you’re trying to reach a general audience who just wants a task done. The shift to Codex as a full desktop product forced a design change. Instead of showing code, the interface now shows a simple pointer that clicks around the screen, sometimes with a picture-in-picture view of the action. The model doing the work underneath is largely the same coding agent architecture. What changed is the presentation: less visibility into the mechanics, more trust that the output is correct. The bigger change was speed. Early computer use was visually impressive but slow enough that using it for real work felt impractical. OpenAI engineers describe a tipping point where the capability became fast enough that people stopped noticing when an agent quietly switched from a direct integration to clicking through a UI instead. The task just finished, and the method became invisible. Why does computer use matter when APIs and MCP integrations already exist? The pitch for MCP (Model Context Protocol) and direct API integrations is efficiency: they’re typically faster and more token-efficient than watching a cursor click through a web page. But that only helps if the integration exists. In practice, most of the systems people deal with daily, government portals, insurance paperwork, pediatrician forms still handled by fax, legacy enterprise software, will never get a clean API or MCP connector built for them.; tools: Codex. Human role: The original post does not say how a person stayed involved.; limitations: Computer use is slower and less token-efficient than direct API integrations, making it suitable only for the long tail of tools without connectors. Reliability depends on visual recognition and model reasoning capabilities. Human approval is critical before submitting any changes or filings, as the source does not specify automated error handling for financial data..
Tools
Published on Agent Work Atlas:
Overview
OpenAI engineers explain how Codex's computer-use feature clicks, browses and fills forms across your desktop, and why it took years to get reliable.
Workflow
OpenAI engineers explain how Codex's computer-use feature clicks, browses and fills forms across your desktop, and why it took years to get reliable. What is computer use in OpenAI’s Codex? Computer use is the capability that lets an AI agent operate a computer the way a person does: moving a cursor, clicking buttons, scrolling pages, and filling in forms inside real applications and websites, rather than calling an API behind the scenes. In OpenAI’s Codex, this started as a developer-facing feature built on coding agents that ran visible scripts. As Codex expanded into a full desktop app for a broader audience, the same underlying capability got a friendlier face: a pointer that visibly clicks around the screen instead of exposing the raw automation script. TL;DR Computer use began as a niche developer tool where Codex showed the actual scripts and selectors it used to click through an app, which worked but wasn’t approachable for non-technical users. The interface changed as the audience grew, replacing visible code with a simple on-screen pointer so people could watch the agent work without needing to understand what’s happening underneath. Reliability crossed a tipping point in the last year or so, moving computer use from what one OpenAI engineer called an “impressive party trick” that was slow to something fast enough to use casually and constantly. Computer use now acts as a universal connector for tasks that have no API or MCP integration, like government paperwork, PDFs, or legacy web forms that will likely never get a dedicated connector. Direct integrations (Slack, email, and similar connectors) still matter because they’re often faster and more token-efficient than clicking through a UI, so OpenAI is investing in both approaches rather than picking one. Internal adoption at OpenAI happened unevenly by function, with coding teams switching first because their work lived in local command-line tools, and other teams like legal following once the model got good enough at reasoning over large amounts of material. The bigger shift is behavioral, according to OpenAI staff: the habit that matters most is catching yourself before doing a task manually and routing it to the agent instead. How did computer use evolve from a party trick to a reliable tool? According to OpenAI engineers who build the Codex desktop app, computer use has existed in some form for a while, but it stayed a developer tool for a long time because it needed technical users who wanted to see the scripts, selectors, and clicks the model was running under the hood. That transparency worked for engineers debugging automation, but it doesn’t hold up when you’re trying to reach a general audience who just wants a task done. The shift to Codex as a full desktop product forced a design change. Instead of showing code, the interface now shows a simple pointer that clicks around the screen, sometimes with a picture-in-picture view of the action. The model doing the work underneath is largely the same coding agent architecture. What changed is the presentation: less visibility into the mechanics, more trust that the output is correct. The bigger change was speed. Early computer use was visually impressive but slow enough that using it for real work felt impractical. OpenAI engineers describe a tipping point where the capability became fast enough that people stopped noticing when an agent quietly switched from a direct integration to clicking through a UI instead. The task just finished, and the method became invisible. Why does computer use matter when APIs and MCP integrations already exist? The pitch for MCP (Model Context Protocol) and direct API integrations is efficiency: they’re typically faster and more token-efficient than watching a cursor click through a web page. But that only helps if the integration exists. In practice, most of the systems people deal with daily, government portals, insurance paperwork, pediatrician forms still handled by fax, legacy enterprise software, will never get a clean API or MCP connector built for them.
Reported result
The source reports an anecdotal success where Codex caught a significant error in a personal tax filing worth thousands of dollars in the user's favor. No measured accuracy rates or broader business impact metrics are provided. Proposed acceptance check: Verify that the agent correctly identifies at least one known error in a sample tax document without human intervention during the review phase.
Human role
The original post does not say how a person stayed involved.
Limitations
Computer use is slower and less token-efficient than direct API integrations, making it suitable only for the long tail of tools without connectors. Reliability depends on visual recognition and model reasoning capabilities. Human approval is critical before submitting any changes or filings, as the source does not specify automated error handling for financial data.
Try it
I want to use OpenAI Codex to review a set of tax documents and fill out a government form. Here are the inputs: [replaceable inputs: path to tax PDFs, URL of government portal, specific form fields to fill]. Please perform a dry run on one sample document first. Navigate to the portal, fill the form based on the PDF data, and stop before submitting. Show me the filled form and any discrepancies found. If the sample is correct, proceed with the remaining documents. Do not submit any forms without my explicit confirmation. If you encounter a CAPTCHA or a field you cannot read, pause and ask for clarification. Measure success by the number of fields correctly populated and errors identified in the dry run.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.