How to Let AI Chatbots Read Your Store (And Why Most Shops Are Accidentally Blocking Them)

Before ChatGPT can recommend your product, it has to be able to read your product page. That sounds obvious.

It is also the single most common reason e-commerce brands are invisible in AI answers, and almost nobody checks it. Most store owners aren't technical founders.

Traffic to your store from AI chatbots is taking up more and more space. Shopify reported that AI-referred visits to its storefronts grew 197%, roughly tripling, year over year in Q2 2026, with orders growing at the same rate.

Semrush's analysis of over a billion lines of clickstream data found ChatGPT's outbound referral traffic climbing 206% year over year between January 2025 and January 2026, even after its own user numbers leveled off.

That growth in AI traffic only reaches stores that AI systems can actually crawl.

The problem: a 2023 decision is still costing you sales in 2026

When ChatGPT first went mainstream, many conscious online publishers and online store owners blocked AI bots on principle. Nobody wanted their content scraped for free.

Those online entities modified their robots.txt rules such that AI crawlers were blocked from learning about them.

In 2026 and beyond, online entities such as ecommerce stores are realizing they should let AI crawlers visit their websites if it means receiving citations and traffic back. GPTBot is still the most blocked AI crawler on the web, appearing in 5.52% of all disallow rules analyzed across Cloudflare's network in Q1 2026, ahead of CCBot at 5.08% and ClaudeBot at 4.88%.

A different study from 2026 found 41% of B2B sites were still blocking at least one major AI bot, mostly leftover settings from that initial block-everything phase back in 2023.

There is a second trap that catches a lot of store owners off guard. Since July 1, 2025, every new domain added to Cloudflare has had major AI crawlers blocked by default at the network level, before your site's robots.txt file is even checked.

Those default settings change again on September 15, 2026, when new domains will have AI training and AI agent crawlers blocked by default on ad-supported pages, while AI search crawlers remain allowed, as they have a better chance of sending you back search traffic.

You might want to customize which bots you give access to and block over time, depending on how the AI situation evolves.

AI training bots and AI search bots are not the same thing

AI training crawlers gather content from your website to train future AI models. GPTBot, ClaudeBot, CCBot, and Google-Extended mostly fit this description.

Search and retrieval crawlers fetch pages to answer live user questions, sharing a link and a potential site visit back to you. OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Amzn-SearchBot work this way.

A single rule blocking all "AI bots" combines two totally different business choices into one, and it usually is the wrong option. Cloudflare's traffic breakdown shows that training makes up nearly 80% of AI bot crawling volume, meaning most AI crawling activity happens in the category that matters least for store visibility.

Blocking training crawlers costs you very little referral traffic. Blocking search/retrieval crawlers cuts you off from the fastest-growing discovery channel in online retail.

The audit: four checks to perform

Read your robots.txt line by line. Go to yourwebsite.com/robots.txt for your website.

Look for any Disallow rules targeting GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, PerplexityBot, Google-Extended, Bytespider, or Meta-ExternalAgent. Make a clear & informed choice on each one instead of keeping settings someone set up years ago.

A reasonable partial robots.txt setup for a store that wants AI recommendations would be:

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ChatGPT-User
Allow: /

Google-Extended controls whether you show up in AI Overviews and AI Mode without impacting your standard Google search rankings, so most retailers have no reason to block it.

Check your CDN (content delivery network) firewall settings, not just robots.txt. A firewall rule (defined in your CDN tool, such as Cloudflare) runs before robots.txt is ever read, meaning a WAF block overrides any "allow" rule in your robots.txt.

If you use Cloudflare, Fastly, or similar services, check the bot management settings directly. This is where most "why is my allow rule ignored?" issues get solved.

Look at 30 days of server logs. Count requests and response codes for OAI-SearchBot, GPTBot, ClaudeBot, PerplexityBot, and Google-Extended. Flag any frequent 403 or 429 errors.

A crawler hit by heavy rate limits is practically the same as a blocked crawler.

Make sure your product pages render server-side. Many AI crawlers struggle with JavaScript.

If your price, stock status, description, or structured data only load client-side, the crawler just sees an empty page. View the raw page source (not the browser inspector) and search for your product name and price. If they aren't in the raw HTML, fix that next.

What success looks like

Once crawlers can read your site, the incoming traffic is well worth the effort, if your store is not completely new. Even if it is new, efforts can pay off in the long run.

Shopify's Q1 2026 retail data showed that AI-referred visits convert at nearly 50% higher rates than regular organic search and feature 14% higher average order values. Plus, over half of AI-referred visits go straight to a product page, compared to around 20% for organic search.

The reason is clear: someone clicking over from an AI assistant has already compared products and narrowed down their choice during personalized conversation with the chatbot. They often arrive on a store ready to buy.

One key detail to keep in mind

A robots.txt file is a polite request, not an absolute barrier. Well-behaved crawlers follow it, but some do not.

Perplexity, for example, has noted that user-requested fetches generally bypass robots.txt because a real person triggered the lookup rather than an automated bot. If you need true content protection, robots.txt won't cut it. Use firewall rules instead.

However, the opposite problem happens far more often. Most stores aren't being over-crawled; they're simply invisible to AI assistants & chatbots.

FAQ

Does allowing AI crawlers hurt my Google rankings?

In general, no. People are using AI assistants more and more, so you are going to want to cash in on that usage by doing everything in your power to make these AI chatbots recommend you instead of your competitors.

However, you might want to block the AI crawlers that are simply training on your data and won't send referral traffic back to you.

If I block GPTBot, am I still visible in ChatGPT?

Possibly, but not as strongly. GPTBot handles training crawls, while OAI-SearchBot handles live searches for ChatGPT Search.

Blocking GPTBot alone doesn't remove you from live search answers, but it does leave your brand out of the training for the model's core knowledge base.

Should I add an llms.txt file?

It's cheap to set up and won't hurt, but no major AI platform has officially adopted it as a standard yet. Don't rely on it as a substitute for a clean robots.txt file and proper server-side rendered HTML.

How long until I see results?

Crawlers need time to revisit and re-index your pages. Most stores notice AI-referred traffic shifting within four to eight weeks after unblocking, assuming the pages can be read easily by AI.

Not sure what your store is currently blocking? We perform crawler-access audits for e-commerce brands as the starting point for every AI visibility project. Email us at info@blueskiesdigital.com and we'll tell you what the bots can and cannot see.