Free SEO tool
Free AI Crawler Checker
Enter your website and see which AI crawlers your robots.txt lets in and which it blocks, grouped by AI search, training and on-demand browsing. If something is blocked by mistake, you get the lines to fix it.
Log in to use this tool
All of our SEO tools are free with an account. Sign up in under a minute, no card needed.
What it checks
What this tool looks at
Seventeen AI crawlers
OpenAI, Anthropic, Perplexity, Google, Apple, Microsoft, Amazon, Meta and others, each with what it is used for.
Search vs training
Blocking training is a choice. Blocking AI search crawlers keeps you out of AI answers. The tool separates the two.
A fix you can paste
If search crawlers are blocked, you get ready-made lines that allow them.
How to use it
Three steps
Enter your website
Type your domain, for example example.com.
We read the rules
We fetch the file and apply its rules to each crawler.
Fix what is blocked
Copy the suggested lines if an AI search crawler is blocked by mistake.
Good to know
- It reads the site’s crawl rules only. A firewall or CDN bot-protection setting can still block a crawler that the rules allow.
- Only whole-site blocks are reported. A rule that blocks one folder does not count as blocked.
- Crawlers can ignore the rules. The file is a request, and well-behaved crawlers follow it.
FAQ
Questions about this tool
Should I allow AI crawlers?
It depends on your goal. If you want to be found and cited in AI answers, allow the AI search crawlers. Training crawlers are a separate choice: you can block them and still be cited by AI search.
Does blocking Google-Extended affect my Google rankings?
No. Google-Extended controls whether your content is used for Gemini training and grounding. It does not affect Google Search crawling or ranking.
Why does it say allowed when my firewall blocks bots?
This tool reads the crawl rules only. Security plugins and CDN settings can block crawlers separately, so check those too.
More free tools
Keep going
Robots.txt & Sitemap Checker
Check that search engines can crawl your site and that your sitemap works.
Meta Tag Checker & Google Preview
Check a page’s title, description and tags, and preview how Google may show it.
Broken Link Checker
Find dead and redirecting links on any page.
llms.txt Generator
Draft a guide to your site for AI assistants, built from your sitemap.
Schema Markup Generator
Create JSON-LD for organizations, businesses, articles, FAQs, products and breadcrumbs.
SEO Intent Analyzer
Type a keyword to see the search intent behind it, with title ideas and a page outline.
Want the full picture?
This tool checks one thing. The free ZIT Solution SEO audit checks your homepage and up to 50 pages: technical SEO, on-page, content, schema, internal links, speed and AI search readiness, with a prioritized fix list.