Order groceries with computer-use agents
OpenAI's Operator was the best model I tried. Task: The author tested OpenAI's Operator agent by providing a hand-written image of a 16-item grocery list. The workflow involved OCR interpretation, navigating to a specific Harris Teeter store website, and adding items to an online cart. The human intervened multiple times to correct misread items, re-login the agent after it was logged out, and resolve confusion about delivery availability.; tools: Operator. Human role: The original post does not say how a person stayed involved.; limitations: Computer-use agents are currently slow, error-prone, and require significant human oversight for complex tasks like grocery shopping. The author suggests these agents may not improve user experience due to their clumsy interface. Reliability issues include session timeouts and incorrect interpretation of visual data or website availability..
Tools
Published on Agent Work Atlas:
Overview
OpenAI's Operator was the best model I tried.
Workflow
The author tested OpenAI's Operator agent by providing a hand-written image of a 16-item grocery list. The workflow involved OCR interpretation, navigating to a specific Harris Teeter store website, and adding items to an online cart. The human intervened multiple times to correct misread items, re-login the agent after it was logged out, and resolve confusion about delivery availability.
Reported result
Operator failed to accurately interpret the shopping list, including unchecked items and misreading quantities. It successfully navigated to the store website but only added 13 of 16 items to the cart, missing pineapple, onions, and pasta sauce. The process took over 30 minutes with human assistance, compared to less than 4 minutes for the author to complete the task manually.
Human role
The original post does not say how a person stayed involved.
Limitations
Computer-use agents are currently slow, error-prone, and require significant human oversight for complex tasks like grocery shopping. The author suggests these agents may not improve user experience due to their clumsy interface. Reliability issues include session timeouts and incorrect interpretation of visual data or website availability.
Try it
I want to test an AI computer-use agent for grocery shopping. I will provide a photo of my handwritten shopping list [replaceable input: image file]. Please use OpenAI Operator to: 1) Extract the checked items from the list. 2) Navigate to the website of my nearest grocery store [replaceable input: store name/location]. 3) Add the extracted items to the online cart. Stop and ask for my confirmation before finalizing the order. If you encounter login issues or unclear items, pause and report the specific error. Do not proceed if the item count differs from my manual count. Measure success by comparing the time taken and accuracy against my manual entry.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.