Turn a docs site into Markdown files with an agent
They used a custom crawler so Claude could save a 60-page docs site as Markdown without crowding the chat.
- What they did
- They created pulpie-mcp, an MCP server with four commands for retrieving, storing, crawling, and enumerating pages. A local extraction model sits behind a backend that launches automatically. With that in place, they asked Claude Code to work through a 60-page documentation site, discover pages from robots.txt and sitemaps, and split the job across four concurrent workers. The pages were written out as Markdown files with frontmatter instead of being returned to the conversation.
- How it went
- The whole run finished in about 49 seconds. Around 10 seconds were spent bringing the backend up, and about 37 seconds were used to crawl and save all 60 pages. It has not been checked on client-rendered pages, which can come back blank, and it does not support PDFs.
- Tips
- This setup uses a model licensed CC BY-NC 4.0, so it is for non-commercial use. It was only tried on Linux with an NVIDIA GPU, and it used about 420 MB of VRAM.
Try it yourself with Claude Code
Use [agent name] to mirror the documentation at [docs URL] into clean Markdown files. Start by finding the site’s pages from robots.txt and any sitemap files, then follow only links that stay within the same documentation site. For each page, save a separate .md file in [output folder], include frontmatter, and remove menus, footers, and other page chrome. Work at a polite pace. If you encounter a step that would overwrite files, delete anything, send data out, or submit a job, stop and ask for my approval first. When you are done, tell me how many pages were saved, which URLs failed, and whether anything looked empty or incomplete.
Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.
Get 3 agent uses like this every morning
With the day's top AI news. Free, and one click to unsubscribe.
Related agent uses
Revive old apps and sketch new ones with a coding agent
Terence Tao used a coding agent to port old Java applets to JavaScript and to build two new math visualizations in conversation.
Check a codebase for redundant packages and polyfills
An agent can scan code and install commands to find dependencies that can be replaced with built-ins or better options.
Connect a coding agent to an Excalidraw board
They hooked an installed coding agent to a live Excalidraw canvas so it could read the scene and edit the drawing directly.
Have coding agents test your onboarding flow
Run coding agents through your product’s onboarding and get a scorecard, fix list, and session traces.
Run coding agents in parallel with tmux and Markdown specs
Manuel Schipper uses separate tmux panes and numbered Markdown specs to keep several coding agents moving at once.
Profile and tune a CUDA kernel with an agent
They used a LangGraph agent to test CUDA kernel tweaks, verify results, and profile candidates with Nsight.