AI COMMERCE GUIDE
How to check whether AI crawlers can access your WooCommerce store.
Crawler access is only one part of AI commerce readiness, but it is the first technical gate. If a discovery system cannot reach the store or product pages, better descriptions and structured data cannot help.
Start with WordPress visibility
Open Settings → Reading and confirm that the site is not asking search engines to avoid indexing it. That setting is useful during development, but leaving it enabled on a live store can undermine discovery across traditional search and newer AI systems.
Also test the storefront while signed out. Product pages must be publicly reachable without an administrator session, maintenance screen, password wall, or geographic challenge that blocks ordinary crawlers.
Read robots.txt by crawler—not as one global answer
A robots.txt file can allow one crawler and block another. Check the rules that apply to the specific user agent you care about, including OAI-SearchBot, GPTBot, OAI-AdsBot, Claude-SearchBot, PerplexityBot, Applebot, and Google-Extended.
These crawlers do not all serve the same purpose. Search retrieval, model training, advertising, and shopping discovery may use different user agents. A store can intentionally allow one and block another; the important part is knowing what the current rules actually do.
Confirm the sitemap exposes products
Open the XML sitemap and confirm that active product URLs are present. A sitemap should return a successful response, use the correct public domain, and avoid sending crawlers to redirects, staging addresses, or products that are no longer available.
The sitemap is a discovery path, not a guarantee. It helps systems locate products, but each product page still needs to be accessible, canonicalized correctly, and useful when fetched.
Test representative product pages
Check a simple product, a variable product, a sale item, and any important category page. Confirm each URL returns a normal public page, points to the intended canonical URL, and exposes the product information without requiring a browser interaction to reveal everything important.
Pay attention to security and optimization plugins. Aggressive bot blocking, country rules, JavaScript challenges, and caching mistakes can affect crawlers differently from human visitors.
Separate access from visibility
An allowed crawler and a successful response only prove that the technical access check passed. They do not prove that the page was crawled, indexed, cited, recommended, ranked, or used in a shopping answer.
That distinction matters because any tool promising guaranteed AI placement from a robots.txt change is selling fantasy. Access removes a blocker; strong products, trustworthy content, complete data, and the discovery system’s own choices still determine what happens next.