Fill out a Form 1040 with computer use
Posted on Dec 5, 2024 • Edited on Dec 7, 2024. Task: Posted on Dec 5, 2024 • Edited on Dec 7, 2024 In a previous discussion, I explored Claude’s Computer Use capabilities with browser automation. In this blog post, I'm giving Claude Computer another try. This time, I want to see if it can fill out a tax return using data I have stored on my hard disk. I'm intentionally not giving it the data in the prompt; instead, I want to see if it can: The Challenges: Read the form and determine the required fields. Find the relevant information to fill in those fields from the data on disk. Fill out the form accurately. For this experiment, I've selected a tax return form, specifically the IRS Form 1040: https://www.irs.gov/pub/irs-pdf/f1040.pdf And to make things easier for Claude (and because I initially had trouble with PDF editing), I've generated a simplified web-based version of the same form: Why the web form and not the PDF editor? I tried using a PDF editor first, but Claude CS didn't manage to fill it out. I suspect this is more of a driver or compatibility issue. So, I switched to a browser-based version for this test. I previously did the same experiment with Formix PDF (see screenshot below), which uses Claude afaik, and the results were not bad. I'm wondering if using Claude Computer could do even better. For this experiment, I created some sample tax data and saved it in the /.antropic folder, which is shared between Claude Computer and my Mac. Here's the script I used to generate the data: Having all the data ready, let's go! The Initial Prompt: Here's the initial prompt I gave to Claude Computer: "fill in the form in firefox using the fields in /.home/computeruse/.anthropic" Initially, it started a bit off-track, asking for information that I had already provided in the prompt. So, I simply told Claude to re-read the initial prompt. Claude Gets to Work: And it did! It figured out what it needed to do. First, it checked all the files in the specified folder: Then, it took some screenshots of the Firefox form... Read more in my Blog Post Top comments (0) For further actions, you may consider blocking this person and/or reporting abuse; tools: Claude computer use. Human role: The original post does not say how a person stayed involved.; limitations: PDF editing failed due to suspected driver or compatibility issues, forcing a switch to a web form. The agent initially ignored prompt instructions, requiring human intervention. Source reliability is secondary, as the post is truncated and lacks final outcome data..
Tools
Published on Agent Work Atlas:
Overview
Posted on Dec 5, 2024 • Edited on Dec 7, 2024
Workflow
Posted on Dec 5, 2024 • Edited on Dec 7, 2024 In a previous discussion, I explored Claude’s Computer Use capabilities with browser automation. In this blog post, I'm giving Claude Computer another try. This time, I want to see if it can fill out a tax return using data I have stored on my hard disk. I'm intentionally not giving it the data in the prompt; instead, I want to see if it can: The Challenges: Read the form and determine the required fields. Find the relevant information to fill in those fields from the data on disk. Fill out the form accurately. For this experiment, I've selected a tax return form, specifically the IRS Form 1040: https://www.irs.gov/pub/irs-pdf/f1040.pdf And to make things easier for Claude (and because I initially had trouble with PDF editing), I've generated a simplified web-based version of the same form: Why the web form and not the PDF editor? I tried using a PDF editor first, but Claude CS didn't manage to fill it out. I suspect this is more of a driver or compatibility issue. So, I switched to a browser-based version for this test. I previously did the same experiment with Formix PDF (see screenshot below), which uses Claude afaik, and the results were not bad. I'm wondering if using Claude Computer could do even better. For this experiment, I created some sample tax data and saved it in the /.antropic folder, which is shared between Claude Computer and my Mac. Here's the script I used to generate the data: Having all the data ready, let's go! The Initial Prompt: Here's the initial prompt I gave to Claude Computer: "fill in the form in firefox using the fields in /.home/computeruse/.anthropic" Initially, it started a bit off-track, asking for information that I had already provided in the prompt. So, I simply told Claude to re-read the initial prompt. Claude Gets to Work: And it did! It figured out what it needed to do. First, it checked all the files in the specified folder: Then, it took some screenshots of the Firefox form... Read more in my Blog Post Top comments (0) For further actions, you may consider blocking this person and/or reporting abuse
Reported result
The source does not report a final successful submission or accuracy metrics. It notes the agent began working after correction but cuts off before showing the final filled form. A measurable acceptance check would be verifying all 1040 fields match source data with zero errors.
Human role
The original post does not say how a person stayed involved.
Limitations
PDF editing failed due to suspected driver or compatibility issues, forcing a switch to a web form. The agent initially ignored prompt instructions, requiring human intervention. Source reliability is secondary, as the post is truncated and lacks final outcome data.
Try it
Act as a tax preparation assistant using Claude Computer Use. My goal is to fill a web-based IRS Form 1040 using data from [replaceable: local folder path]. Steps: 1. List files in [replaceable: local folder path] to identify W-2, 1099, and deduction receipts. 2. Open [replaceable: URL of web form] in Firefox. 3. Extract required fields from the form and map them to data in the files. 4. Fill the form fields one by one. 5. Stop before submitting. Take a screenshot of the completed form. Safeguards: - Do not submit or save the form. - If a field is ambiguous, ask me for clarification. - Verify each entry against the source file before typing. Deliverable: A screenshot of the filled form and a list of any fields left blank due to missing data. Confirm with me before proceeding to any further actions.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.