Crawl documentation into clean Markdown for AI agents
An MCP server that gives AI agents their own tools for turning web pages into clean Markdown. Task: An MCP server that gives AI agents their own tools for turning web pages into clean Markdown. It runs the Pulpie content extraction model locally, so tables, code blocks, links, and images come through intact instead of getting summarized away. The agent can read a page inline, save it as a .md file, crawl a whole docs section into a folder, and check what it already saved before fetching again. Why I made it I already had a little local UI for Pulpie that I used to pull docs and articles into Markdown, and it works really well. But I was still the one doing the pulling. Most agent fetch tools run pages through a small model that summarizes or trims them, which is not what you want when the agent needs the actual API reference. So this gives the agent the same thing I was using. It can…; tools: custom-built or unnamed agent. Human role: The original post does not say how a person stayed involved.; limitations: The tool has not been tested on PDFs or completely client-rendered pages, which may return empty results. It was only tested on Linux with NVIDIA/CUDA. The pulpie-orange-small model is licensed CC BY-NC 4.0, restricting commercial use. The author has not evaluated prompt injection risks in the extracted Markdown. The backend automatically shuts down after 30 minutes of inactivity to release VRAM..
Tools
custom-built or unnamed agent
Published on Agent Work Atlas:
Overview
An MCP server that gives AI agents their own tools for turning web pages into clean Markdown.
Workflow
An MCP server that gives AI agents their own tools for turning web pages into clean Markdown. It runs the Pulpie content extraction model locally, so tables, code blocks, links, and images come through intact instead of getting summarized away. The agent can read a page inline, save it as a .md file, crawl a whole docs section into a folder, and check what it already saved before fetching again. Why I made it I already had a little local UI for Pulpie that I used to pull docs and articles into Markdown, and it works really well. But I was still the one doing the pulling. Most agent fetch tools run pages through a small model that summarizes or trims them, which is not what you want when the agent needs the actual API reference. So this gives the agent the same thing I was using. It can…
Reported result
In a test, Claude crawled 60 documentation pages in approximately 49 seconds, including a 10-second backend cold start and 37 seconds for crawling and saving. The output consisted of clean Markdown files with frontmatter metadata (source URL, title, fetch time) stored on disk, allowing the agent to search and reference them without loading all content into the context window. The author reported that the extraction preserved tables, code blocks, links, and images intact.
Human role
The original post does not say how a person stayed involved.
Limitations
The tool has not been tested on PDFs or completely client-rendered pages, which may return empty results. It was only tested on Linux with NVIDIA/CUDA. The pulpie-orange-small model is licensed CC BY-NC 4.0, restricting commercial use. The author has not evaluated prompt injection risks in the extracted Markdown. The backend automatically shuts down after 30 minutes of inactivity to release VRAM.
Try it
I want to set up a local documentation crawler for my AI coding agent using the pulpie-mcp server. First, install the tool using `uv tool install git+https://github.com/pinkpixel-dev/pulpie-mcp` and add it to your agent's MCP configuration (e.g., `claude mcp add --scope user pulpie -- pulpie-mcp`). Ensure you have a compatible GPU (NVIDIA/CUDA on Linux) and accept the CC BY-NC 4.0 license for the Pulpie model. Task: Crawl the documentation for [replaceable input: e.g., 'FastAPI'] into a local folder named [replaceable input: e.g., './docs/fastapi']. Steps: 1. Use the `crawl_docs` tool to fetch the documentation. 2. Verify that the files are saved with frontmatter metadata. 3. Use `list_library` to confirm the saved content. 4. Perform a dry run on a single page using `fetch_markdown` to check quality before full crawling. 5. Review the output for any missing content or formatting issues. Success Criteria: The documentation is saved as clean Markdown files with intact tables and code blocks. The agent can reference these files without loading them all into context. Constraints: Do not use for commercial projects due to model licensing. Handle potential empty results from client-rendered pages by checking the output. If the backend fails to start, verify GPU availability and VRAM usage.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.