When a shopper asks ChatGPT for the finest British knitwear, this heritage house isn't in the room.
Not because the product isn't the best — it is. Because the front door is locked to the machines that now do the recommending. We tested it. Here's exactly what an AI shopping agent gets when it tries to read your site.
PREPARED FOR Sample · anonymised · SUBJECT the-brand.com · CATEGORY British merino knitwear
Agentic readiness
24/100
Invisible
The one-line verdict
The site actively returns 403 Forbidden to OpenAI's crawler. The fastest-growing discovery channel in retail can't fetch a single product — so it recommends someone else.
What the shopper sees
240 years of British craft. A Royal Warrant. 19-micron extra-fine merino. A beautiful, sale-fronted store.
What the agent extracts
GET the-brand.com
> HTTP 403 Forbidden
> 59 bytes. "Forbidden."
> 0 products. 0 prices. 0 specs.
The evidence — what we actually observed
GPTBot (OpenAI)
OpenAI's crawler — the index behind ChatGPT — is refused at the door. 59-byte "Forbidden" response, no content.
403 BLOCKED
ClaudeBot · PerplexityBot
Anthropic's and Perplexity's agents get the same treatment. The three biggest answer engines are all locked out.
403 BLOCKED
Generic crawler
A standard non-JavaScript fetch hits a Vercel Security Checkpoint — a JS challenge wall. Agents that don't run a full browser get the spinner, never the catalogue.
JS WALL
llms.txt · sitemap.xml
A standard request for an agent-readable catalogue guide returned nothing usable — no llms.txt, and the sitemap sat behind the same wall. Nothing tells an AI where the products are.
NONE FOUND
METHOD · Live HTTP requests to the-brand.com on 22 Jun 2026 using the published user-agents of GPTBot, ClaudeBot and PerplexityBot, plus a standard non-JS client. Results are reproducible — we'll re-run any of them live on a call.
The gaps costing you the recommendation
✕You're blocking the buyers, not just the scrapers. Refusing GPTBot removes you from ChatGPT's product knowledge entirely. There's a real distinction between blocking model-training crawlers and blocking the shopping agents that send ready-to-buy customers — right now you're doing both.
✕Your catalogue lives behind JavaScript. Even an agent that gets past the door finds product data rendered client-side, not in the HTML. No server-side facts means nothing to extract.
✕No structured product data. No schema.org/Product markup means price, availability, material, gauge and care instructions aren't machine-readable — so agents can't answer "is it machine washable?" or "how warm is it?" with your product.
✕Your copy is poetry, not answers. Heritage prose wins human hearts; agents need the literal questions answered — "non-itchy merino," "warmest jumper for winter," "true to size."
Instant fixes — the first three move the needle this week
→Allowlist the answer engines at the edge. Let OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended through your WAF/Vercel config. One rule change ends the 403.
→Server-render product pages (or pre-render for bots) so the HTML carries names, prices and specs without needing JavaScript.
→Add schema.org/Product JSON-LD to every PDP — name, price, availability, material, gauge, care. The single highest-leverage thing for agent extraction.
→Publish an llms.txt and a clean product sitemap so agents can discover and map the catalogue.
→Rewrite a handful of PDPs as answer-ready — lead with the questions shoppers actually ask an AI, then the heritage.
The window is now — before the AW26 peak.
Knitwear demand and gift-led search both climb from September into Black Friday and December — exactly when agent-led shopping spikes hardest. Crawlers also need time to re-index after you open the door; a fix made in summer is earning recommendations by peak. A fix made in November is too late for the season that matters most.
Why beyond.partners
We help heritage retailers ship AI — and agentic commerce is where we live. This entire teardown was produced by fynd, our readiness engine, from real live testing of your own site. We don't hand you a PDF and leave; we make the edge change, ship the structured data, and watch the agents start reading you. The brands that get read get recommended. The rest get skipped.
20 minutes. We'll re-run every test live.
You watch a real AI agent try to read your site, then watch what changes the moment we open the door. No deck — just the teardown, live.
Book the live teardown