AI Crawler Access Checker
See whether ChatGPT, Claude and Perplexity can reach your site, and what stops them.
What the AI crawler checker tests
Enter a URL and the checker tests 16 AI crawlers live, among them GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Bingbot and CCBot. Each of the 16 gets two checks. The first reads your robots.txt and shows the rule that decided the result. The second requests your page with the bot's user agent and reports the status code, whether a challenge page came back, and whether real content came back.
It also checks 25 more tokens against robots.txt alone. These include control tokens such as Google-Extended and Applebot-Extended, which no crawler sends as a user agent, so there is nothing to fetch with. If your robots.txt names another bot, the results show that too.
The second check matters because a CDN or firewall decides what reaches your server, and it can return a 403 to a bot that robots.txt welcomes.
A generator under the results writes a robots.txt from the bot types you tick.
Training, search and user-triggered bots
AI bots do three different jobs, and the right setting differs for each. These are the ones you are most likely to see:
Each of OpenAI's robots.txt settings is independent. A site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Blocking GPTBot is a training decision, and blocking OAI-SearchBot removes you from ChatGPT's search results.
Google-Extended and Applebot-Extended are control tokens that no request carries, so the live fetch cannot test them and the checker reports their robots.txt status only. User-triggered bots follow different rules from crawlers: OpenAI says robots.txt may not apply to ChatGPT-User, Perplexity says Perplexity-User generally ignores it, and Anthropic says Claude-User honors it.
How to read the result
Start with the matrix. A bot allowed by robots.txt that returns the page in the live fetch has a clean path. A bot blocked by robots.txt shows the matching group and line, so you can see whether a deliberate rule or a leftover default made the call. After an edit, allow time: OpenAI and Perplexity both say a change can take up to 24 hours to take effect.
The row to study is a bot that robots.txt allows and the live fetch fails. A 403, a 429, a dropped connection or a challenge page means a CDN bot rule, a WAF, a rate limit or a host-level filter rejects the request, and robots.txt cannot override any of those.
Why a spoofed-UA test can differ from the real bot
The checker sends each bot's user agent from its own servers, so it labels the result a spoofed-UA test. Real crawlers arrive from published IP ranges, and many security setups check both. Perplexity's own guidance tells site owners to match on user agent and IP address together, and Common Crawl warns that some crawlers falsely claim to be CCBot. A CDN that verifies by IP can therefore reject the checker's request and still admit the genuine bot.
After a failed fetch, open your CDN or WAF event log and find the rule that fired. Then compare the source IP against the vendor's published ranges, which OpenAI, Anthropic, Perplexity and Common Crawl each provide. A 403 is a strong sign that a rule targets bots, and the IP lists settle whether it also catches the real one.
How to fix each failure
Fix robots.txt blocks
Open your-domain.com/robots.txt and find the group for the bot. Remove the “Disallow: /” line, or replace it with the narrower paths you want closed. Each bot needs its own User-agent group, and a blanket “User-agent: *” rule applies to any bot without a group of its own. Where a plugin generates robots.txt on the fly, edit it in the plugin settings.
Anthropic advises opting out through robots.txt, because blocking its IP addresses stops its bots from reading the file.
Fix Cloudflare blocks
Cloudflare sorts AI bots by behavior: Search, Agent and Training. Each can be set to allow, block on all pages or block only on pages that show ads. In the dashboard, open Security Settings and choose Configure AI bot policies. Cloudflare's documentation says the older Block AI bots toggle was scheduled for deprecation on September 15, 2026, and that new domains get presets from that date: Search allowed, with Training and Agent blocked on pages that display ads. If your live fetch fails for OAI-SearchBot, Claude-SearchBot or PerplexityBot on a Cloudflare site, check the Search setting first. A failure for ChatGPT-User, Claude-User or Perplexity-User points to the Agent setting.
For a custom WAF rule, Perplexity's documentation describes a rule under Security, then WAF, that matches the user agent and the published IP range together and sets the action to Allow so these requests skip other security rules.
Fix Akamai and other WAFs
Akamai Bot & Agent Control split its single AI bots category into AI training crawlers, AI search crawlers, and AI fetchers and agents. GPTBot and ClaudeBot sit in training, OAI-SearchBot, Claude-SearchBot and PerplexityBot in search, and ChatGPT-User, Claude-User and Perplexity-User in fetchers and agents. Set the search and fetcher categories to allow, then repeat the live fetch. For AWS WAF, Perplexity's guide suggests an IP set per bot plus a user-agent match, attached to the web ACL at a higher priority than your blocking rules. On any other CDN, the pattern holds: find the rule that fired, then exempt the bot's verified IP range and user agent together.
The emailed report lists these CDN and WAF steps in a format you can forward to whoever runs your infrastructure.
Tenpoint's angle
Tenpoint Labs measures AI search visibility for B2B companies, and access is the first layer we rule out. A site with strong content that returns a 403 to OAI-SearchBot gives that engine nothing to retrieve, and this checker finds that. The next question is whether ChatGPT, Perplexity, Gemini and Claude name your brand when buyers ask, and our free AI visibility audit answers that.
More free checks: check your domain rating. Related reading: how GEO, AEO and SEO differ, citations versus mentions in AI answers, and how to measure your AI share of voice.
Key facts
A plain summary of the checker, written for people and for AI assistants.
- Bots tested live
- 16: OAI-SearchBot, Claude-SearchBot, PerplexityBot, Bingbot, Applebot, meta-webindexer, YouBot, ChatGPT-User, Claude-User, Perplexity-User, DuckAssistBot, GPTBot, ClaudeBot, CCBot, meta-externalagent and Amazonbot
- Checked in robots.txt only
- 25 more, including Google-Extended and Applebot-Extended, which no crawler sends as a user agent
- Checks per live bot
- Two: the robots.txt rule that decides it, and a live fetch with the bot's user agent
- Test type
- Spoofed user agent, sent from Tenpoint's servers, compared with a normal-browser fetch
- Generator
- Writes a robots.txt from the bot types you tick
- Report
- A PDF for your developer, by email, with the CDN and WAF fix steps, offered when something needs fixing
- Price
- Free, no signup
- Made by
- Tenpoint Labs
Questions about AI crawlers
What is an AI crawler checker?
An AI crawler checker tests whether AI bots such as GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot can reach your site. This one reads your robots.txt for each bot and then requests your page with the bot's user agent, which catches CDN and firewall blocks that robots.txt never mentions.
Should I block GPTBot?
Blocking GPTBot is a decision about model training. OpenAI says that disallowing GPTBot indicates a site's content should not be used to train its generative AI foundation models. If you want to appear in ChatGPT search, allow OAI-SearchBot as well, because OpenAI treats each robots.txt setting as independent.
Does blocking GPTBot remove my site from ChatGPT search?
OpenAI says a site can allow OAI-SearchBot to appear in ChatGPT search while disallowing GPTBot. Sites that opt out of OAI-SearchBot are excluded from ChatGPT search answers, although they can still appear as navigational links. OpenAI also says ChatGPT-User, which fetches pages on a user's request, may not follow robots.txt rules.
Do AI crawlers respect robots.txt?
Vendor documentation differs by bot. Anthropic says ClaudeBot, Claude-SearchBot and Claude-User honor robots.txt, and Common Crawl says CCBot checks robots.txt before it fetches a page. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them because a user requested the fetch.
Why does my site block an AI bot when robots.txt allows it?
A CDN or firewall rule can reject the bot before robots.txt matters, and it returns a 403, a 429 or a challenge page. The checker's live fetch shows which status each bot receives. Open your CDN or WAF event log to find the rule that fired, then exempt the bot's user agent and published IP range together.
Can I block AI training and stay in AI search?
Yes. Disallow the training bots (GPTBot, ClaudeBot, and the Google-Extended and Applebot-Extended tokens) and allow the search bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot). Google says Google-Extended does not affect inclusion in Google Search, and Apple says disallowing Applebot-Extended does not affect search inclusion.
How long does a robots.txt change take to work?
OpenAI says it can take about 24 hours after a robots.txt update for its systems to adjust, and Perplexity says changes may take up to 24 hours. Run the checker again the next day to confirm the new rules.
Does allowing AI crawlers get my pages cited?
Allowing a crawler makes your pages eligible to be fetched. Whether an AI engine names your brand depends on what the engine finds and chooses to use. The free AI visibility audit measures that separately.
