Audit cleaning sites for bot access and page tags
An agent checked 18 cleaning-company sites for robot rules and page metadata, then compared which ones loaded normally and which ones did not.
- What they did
- They began with domains containing “cleaning” from Tranco’s top-1M list and manually trimmed that set to 18 service companies using page titles. After that, they used two Site Check tools on each site: one examined crawler permissions plus response behavior, and the other looked for 8 metadata tags on the homepage.
- How it went
- Nine sites served their homepage. Among those nine, one shut out 12 of the 14 crawlers, seven were missing a share image, and only two had all 8 tags. The other nine sites came back as a 403, a CDN failure, or no response at all.
- Tips
- A blocked or failed request does not prove what an AI bot would actually see, and CDN or firewall rules can override what robots.txt says. Use this kind of check as a snapshot, not a final verdict.
Try it yourself with Claude Code
Here are [number] cleaning-company domains: [paste domains]. For each one, check the site’s crawler rules, response headers, and redirect path for major AI bots including GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Then review the homepage for [list the 8 tags you want checked] and record whether the site serves its homepage, denies access, shows a CDN error, or gives no reply. Stop before any action that would send, delete, pay for, or submit anything, and ask me to confirm. End with a table and a short note on the main missing items.
Fill in the [brackets]. Agents can make mistakes, so review anything before it is sent, paid or deleted.
Get 3 agent uses like this every morning
With the day's top AI news. Free, and one click to unsubscribe.
Related agent uses
Set up a voice agent for round-the-clock support calls
Robin Delta used a voice agent to answer support calls, handle basic issues, and book appointments any time of day.
Audit Facebook ads and propose one change each day
A custom agent checked a client’s Facebook ads every morning, verified leads against CRM records, and suggested one action for approval.
Set up an appointment agent for calls and chats
A clinic used an AI agent to answer, book, and move appointments across phone, text, and web chat.
Connect a voice receptionist to your CRM with MCP
They wired a voice agent to a custom MCP server so it could list, book, and cancel appointments through their CRM.
Use a voice agent to answer business calls
Small businesses set up voice agents to screen calls, capture details, and route real customers to staff while logging everything in their CRM.
Let a voice agent handle service calls and owner updates
A call-answering agent for home-service companies logs key details and sends the owner a text recap.