Many websites block AI crawlers without knowing it — often a blanket disallow from an old robots.txt template. Block GPTBot and you vanish from ChatGPT's knowledge. Check it in 5 seconds.
Full access: keywords, content, AI visibility, backlinks, leads — all fully automated.
Start for freeBrands and agencies worldwide trust RankWay




















ChatGPT, Claude, Perplexity and Google AI answer millions of buying questions every day. Their crawlers — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and 40+ more — decide which websites appear in those answers. Block them in robots.txt and you're neither absorbed into model knowledge nor cited live when a user asks at the decisive moment.
The three crawler types work differently: training crawlers feed the models' permanent knowledge, search crawlers build the AI search index, and live crawlers fetch your page exactly when a user is asking. A single blanket "Disallow: /" from an old template makes you invisible to all three. An llms.txt additionally helps AI systems understand your most important content.
We fetch your website's robots.txt and evaluate which of the known AI crawlers (ChatGPT, Claude, Perplexity, Google AI, Meta, Apple and more) may read your homepage, which are partially restricted and which are fully blocked. We also check whether an llms.txt exists.
It's a trade: protection versus visibility. Block AI crawlers and you won't appear in AI answers — exactly where more and more customers research before buying. For most businesses the visibility loss clearly outweighs the protection.
Allow the relevant user agents in your robots.txt (or remove the blanket disallow) and add an llms.txt. RankWay does both automatically and then measures how often AIs actually recommend you.