Audit cleaning-company sites for AI crawler access
By OpenKrill. Task: By OpenKrill. Published 2026-10-11. Every number here comes from our own Site Check runs on 2026-10-10 at about 22:00 GMT+7. 9 sites read once is a small snapshot, not a survey, so read the numbers that way. How we picked the sites We took every domain with "cleaning" in its name from the Tranco top 1 million list, a public ranking of busy domains. We left out names that looked like registries, forums, supply shops or non-English markets, which gave 28 domains. From the page titles, we then dropped 10 more: 1 software seller, 2 product makers, 1 supply shop, 1 review blog, 1 empty blog, 2 non-English sites, 1 gambling page and 1 basketball site. That left 18 cleaning firms that sell a service. We do not name any of the 18 here. For each site we called 2 Site Check tools. check_ai_crawler_access reads robots.txt, headers and redirects for 14 named AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot. check_page_tags reads the page tags. We counted 8 of them: meta description, canonical link, mobile viewport, og:title, og:description, og:image, og:url and twitter:card. What we found The one site that blocks AI crawlers shuts out 12 of the 14 we check: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Amazonbot. It still lets in the other 2 we check, Bytespider and CCBot. That covers both the bots that collect training data and the ones that fetch a page when a person asks an assistant a question (OpenAI, Overview of OpenAI crawlers). That can be a choice. If it is not your choice, it is a change in robots.txt (RFC 9309, Robots Exclusion Protocol). The 9 sites that sent nothing back A 403 is a refusal of our server. A CDN error or no answer means the site did not serve us, and we cannot tell why. Neither tells us what GPTBot or ClaudeBot gets, because a site can treat each bot by its own rule. So we do not count those 9 as blocked. The reverse is true too: a firewall or CDN rule can turn a bot away even when robots.txt lets it in, and a robots.txt check cannot see that. What to check on your own site Open your robots.txt and look for Disallow: / under a bot name you did not mean to block. Then check your home page for a title, a meta description and a canonical link, which tell search engines what the page is and which address is the real one. Add og:title, og:description and og:image too: they decide what people see when someone shares your link (The Open Graph protocol). None of this makes an AI assistant name you. Nobody can promise that. It only makes sure the door is not shut by mistake. Run the same check on your site: Site Check is free, with no sign-up and no key. Open Site Check, or add it in Claude, ChatGPT or Cursor, then ask "can AI crawlers read example.com?". If you would rather get a written report with the fixes in order, we do that as a fixed-price AI search readiness audit. Sources Tranco list: a research ranking of the top 1 million domains, downloaded 2026-10-07. Koster, Illyes, Zeller, Sassman. RFC 9309, Robots Exclusion Protocol (2022): how robots.txt groups and Disallow rules work. OpenAI, Overview of OpenAI crawlers: GPTBot collects training data, OAI-SearchBot and ChatGPT-User fetch pages for search and for a user's question. The Open Graph protocol: the og:title, og:description and og:image tags a link preview reads. Our own runs: Site Check check_ai_crawler_access and check_page_tags on 18 cleaning-firm sites, 2026-10-10 about 22:00 GMT+7.; tools: custom-built or unnamed agent. Human role: The original post does not say how a person stayed involved.; limitations: The study is a small snapshot of 9 accessible sites, not a comprehensive survey. Results do not account for firewall or CDN rules that might block bots despite permissive robots.txt. The author notes that fixing tags does not guarantee AI assistant visibility, only that the door is not mistakenly shut..
Tools
custom-built or unnamed agent
Published on Agent Work Atlas:
Overview
By OpenKrill.
Workflow
By OpenKrill. Published 2026-10-11. Every number here comes from our own Site Check runs on 2026-10-10 at about 22:00 GMT+7. 9 sites read once is a small snapshot, not a survey, so read the numbers that way. How we picked the sites We took every domain with "cleaning" in its name from the Tranco top 1 million list, a public ranking of busy domains. We left out names that looked like registries, forums, supply shops or non-English markets, which gave 28 domains. From the page titles, we then dropped 10 more: 1 software seller, 2 product makers, 1 supply shop, 1 review blog, 1 empty blog, 2 non-English sites, 1 gambling page and 1 basketball site. That left 18 cleaning firms that sell a service. We do not name any of the 18 here. For each site we called 2 Site Check tools. check_ai_crawler_access reads robots.txt, headers and redirects for 14 named AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot. check_page_tags reads the page tags. We counted 8 of them: meta description, canonical link, mobile viewport, og:title, og:description, og:image, og:url and twitter:card. What we found The one site that blocks AI crawlers shuts out 12 of the 14 we check: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended, Meta-ExternalAgent and Amazonbot. It still lets in the other 2 we check, Bytespider and CCBot. That covers both the bots that collect training data and the ones that fetch a page when a person asks an assistant a question (OpenAI, Overview of OpenAI crawlers). That can be a choice. If it is not your choice, it is a change in robots.txt (RFC 9309, Robots Exclusion Protocol). The 9 sites that sent nothing back A 403 is a refusal of our server. A CDN error or no answer means the site did not serve us, and we cannot tell why. Neither tells us what GPTBot or ClaudeBot gets, because a site can treat each bot by its own rule. So we do not count those 9 as blocked. The reverse is true too: a firewall or CDN rule can turn a bot away even when robots.txt lets it in, and a robots.txt check cannot see that. What to check on your own site Open your robots.txt and look for Disallow: / under a bot name you did not mean to block. Then check your home page for a title, a meta description and a canonical link, which tell search engines what the page is and which address is the real one. Add og:title, og:description and og:image too: they decide what people see when someone shares your link (The Open Graph protocol). None of this makes an AI assistant name you. Nobody can promise that. It only makes sure the door is not shut by mistake. Run the same check on your site: Site Check is free, with no sign-up and no key. Open Site Check, or add it in Claude, ChatGPT or Cursor, then ask "can AI crawlers read example.com?". If you would rather get a written report with the fixes in order, we do that as a fixed-price AI search readiness audit. Sources Tranco list: a research ranking of the top 1 million domains, downloaded 2026-10-07. Koster, Illyes, Zeller, Sassman. RFC 9309, Robots Exclusion Protocol (2022): how robots.txt groups and Disallow rules work. OpenAI, Overview of OpenAI crawlers: GPTBot collects training data, OAI-SearchBot and ChatGPT-User fetch pages for search and for a user's question. The Open Graph protocol: the og:title, og:description and og:image tags a link preview reads. Our own runs: Site Check check_ai_crawler_access and check_page_tags on 18 cleaning-firm sites, 2026-10-10 about 22:00 GMT+7.
Reported result
Of the 18 sites, 9 returned no content (403 or CDN errors), 4 refused the server, and 5 had CDN errors. Among the 9 accessible sites, only 1 blocked AI crawlers in robots.txt, shutting out 12 of 14 tested bots. Only 2 of the 9 accessible sites had all 8 checked tags present, while 7 lacked og:image and 5 lacked og:title/description.
Human role
The original post does not say how a person stayed involved.
Limitations
The study is a small snapshot of 9 accessible sites, not a comprehensive survey. Results do not account for firewall or CDN rules that might block bots despite permissive robots.txt. The author notes that fixing tags does not guarantee AI assistant visibility, only that the door is not mistakenly shut.
Try it
I need you to audit [replaceable: list of 5-10 competitor URLs] for AI crawler readiness. Use a tool or script to check robots.txt for blocks against GPTBot, ClaudeBot, and PerplexityBot. Also, verify the presence of meta description, canonical link, og:title, og:description, and og:image on the homepage. First, run this check on [replaceable: one sample URL] and show me the raw output. Do not proceed to the full list until I confirm the format is correct. If a site returns a 403 or timeout, mark it as 'Inaccessible' rather than 'Blocked'. Output a table with columns: URL, AI Crawler Status (Allowed/Blocked/Inaccessible), Missing Tags, and Notes. Ensure no changes are made to the websites; this is read-only analysis.
Source reports are not independently verified. Publication date is the catalog publication date, not the original article date.