Let agents fill out web forms using text-grid page rendering
Hi HN! Task: The author developed TextWeb to reduce token costs for AI agents filling job applications by replacing vision-based screenshot analysis with structured text grids. Headless Chromium renders pages, and TextWeb maps visible elements, positions, and interactivity onto a character grid preserving spatial layout. Interactive elements receive reference numbers (e.g., [3]Click me) allowing agents to execute actions like 'click 3' or 'type 7 hello' without vision models. The tool integrates via MCP, function-calling definitions, LangChain, CrewAI, HTTP API, CLI, and Node.js.; tools: custom-built or unnamed agent. Human role: The original post does not say how a person stayed involved.; limitations: Prerequisites include installing TextWeb via npm and configuring integrations with LLM frameworks or MCP servers. The source does not specify operating costs, error handling for complex dynamic pages, or limitations on grid fidelity for non-standard UIs. Human approval workflows are not described, leaving uncertainty about safety controls for automated form submissions..
Tools
custom-built or unnamed agent
Published on Agent Work Atlas:
Overview
Hi HN!
Workflow
The author developed TextWeb to reduce token costs for AI agents filling job applications by replacing vision-based screenshot analysis with structured text grids. Headless Chromium renders pages, and TextWeb maps visible elements, positions, and interactivity onto a character grid preserving spatial layout. Interactive elements receive reference numbers (e.g., [3]Click me) allowing agents to execute actions like 'click 3' or 'type 7 hello' without vision models. The tool integrates via MCP, function-calling definitions, LangChain, CrewAI, HTTP API, CLI, and Node.js.
Reported result
The source does not report measured outcomes, performance metrics, or specific success rates for the TextWeb implementation. It only states the motivation was reducing token burn from vision models and describes the technical approach. No evidence confirms actual execution results or business impact beyond the tool's design intent.
Human role
The original post does not say how a person stayed involved.
Limitations
Prerequisites include installing TextWeb via npm and configuring integrations with LLM frameworks or MCP servers. The source does not specify operating costs, error handling for complex dynamic pages, or limitations on grid fidelity for non-standard UIs. Human approval workflows are not described, leaving uncertainty about safety controls for automated form submissions.
Try it
Act as an automation engineer implementing TextWeb for job application forms. First, install TextWeb globally using 'npm install -g textweb'. Select one target job application URL and run TextWeb to generate a text-grid representation. Verify that interactive elements have reference numbers and spatial layout is preserved. Configure your LLM agent (e.g., via MCP or function-calling) to interpret the grid and propose actions like 'click [3]' or 'type [7] [value]'. Before executing any real submission, perform a dry run on a sample form and present the proposed actions to me for approval. Only proceed with actual form filling after I confirm the actions are correct. If the grid is unreadable or elements lack references, report the error and do not attempt to submit. Provide the text-grid output and the agent's proposed action sequence for my review.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.