Some tools and agencies now give you a score for how often AI assistants like ChatGPT mention your store. Before you trust one, ask how many questions it is based on and how many times they were asked. New data this month shows a small set of questions can move the score by 20 or 30 points with nothing changing on your site. The fix is to track enough questions, to pay more attention to whether your brand is mentioned than to which page gets linked, and to look at each assistant separately.
A small sample gives you a jumpy score
OtterlyAI, a company that sells AI visibility tracking, asked seven AI assistants the same 520 questions every day for up to three months, across three US product categories. That came to just over 250,000 answers. Two findings matter for a store owner.
The first is that brand mentions are much steadier than links. Whether an assistant mentions your brand stays fairly stable from day to day, but the web pages it links to as sources change a lot. Brands that showed up once in a month and never again were fairly rare (12-19%), while one-off links were much more common (26-67%). ChatGPT and Google's AI Mode kept only about a quarter of the previous day's links. Perplexity kept about three quarters.
The second is that the number of questions you track matters. With only 10 questions, OtterlyAI found that the share of answers mentioning a brand could land anywhere from 39% to 71% on luck alone. That is a 30-point spread. With 50 questions the range narrowed to 49-62%, and with 100 questions to 51-60%.
There are limits to this. The categories are American, the question-count test used one brand's data, and OtterlyAI has an obvious interest in you tracking more questions. The overall point still stands. Checking a handful of questions once doesn't tell you much.
If you or an agency are tracking how AI assistants see your store, make "how often your brand is mentioned" the main figure. Treat the links as a clue to why. On a small set of questions, don't read much into a move of a few points from one month to the next. And a screenshot of a single ChatGPT answer only shows you what it said that one time.
The assistants don't agree on where to look
OtterlyAI's September update, drawn from 16 US industry reports, shows how differently the assistants behave. Between 13 and 15 September, Google's AI Mode cut YouTube from about 11% of the sources it linked to, down to around 3.5%. Links to LinkedIn rose on five of the seven assistants. ChatGPT now links to Reddit about 94% less often than it did in July, and Claude didn't link to Reddit once in the 77 days measured. Across all of them, brand and company websites still made up about 58% of the links.
So something that works on one assistant can do nothing on another, and the mix can change within days. That is a good reason to keep your own website in good shape, because that is still where most of the links go. Be wary of anyone selling a quick fix aimed at a single assistant, and ask for any visibility report to show each assistant on its own line.
Old guides still work if you keep them up to date
In a separate experiment on its own blog, OtterlyAI changed only the date shown on 10 older articles. Over the next 30 days, AI assistants linked to those articles 31.5% more often, while two articles it left alone were linked 10.6% less. It is a small test: one website, a software topic and only two articles to compare against. OtterlyAI itself says not to change dates without reviewing the content.
If you have buying guides, care guides or how-to articles that still bring in sales, put them on a review schedule. Fix anything out of date, then show the date you last checked them. Shopify blog posts have a published date and an updated date your theme can show, but Shopify changes the updated date every time you save the post, even for a typo. Adding your own "last reviewed" date is more honest. We wouldn't change the date without changing the content.
A free look at how Microsoft's AI uses your pages
Most AI visibility figures are estimates, made by sending questions to the assistants and recording the answers. Microsoft is the exception. Bing Webmaster Tools, Microsoft's free dashboard for website owners, now has an AI Performance report. It shows how often your pages are quoted as sources in Copilot and in Bing's AI answers, and the searches Copilot ran to find them. A June update added topics, the kind of question being asked, and your share of the sources quoted. It only covers Microsoft's assistants and is still marked as a preview, but it is free.
If your store isn't set up in Bing Webmaster Tools yet, it is worth ten minutes this week. If you already use Google Search Console, you can bring your site across from there in a few clicks.
What we're watching
- Whether an independent study finds the same effect from the number of questions. So far the evidence comes from one vendor.
- Whether Google's AI Mode keeps linking to YouTube less. OtterlyAI had only five days of data after the change.
- How Bing's AI Performance figures look for New Zealand stores with modest traffic. We haven't tried it on a client account yet.
Setting up a reliable starting measurement like this for your own store is part of what an AI readiness audit does.
Sources
- OtterlyAI (Thaylise Nakamoto), "We Measured How Stable AI Citations and Brand Mentions Really Are (and Why 10 Prompts Isn't Enough)", 11 Sep 2026 - https://otterly.ai/blog/ai-search-visibility-stability/
- OtterlyAI (Rick Tousseyn), "The September Update: The Month AI Search Stopped Agreeing About Social", 22 Sep 2026 - https://otterly.ai/blog/ai-search-updates-september/
- OtterlyAI (Thaylise Nakamoto), "We Updated Only the Publication Date on 10 Articles. AI Citations Rose 31.5%", 24 Sep 2026 - https://otterly.ai/blog/publication-date-ai-citations-experiment/
- Shopify, Liquid reference, "article" object - https://shopify.dev/docs/api/liquid/objects/article
- Microsoft Bing, "Introducing AI Performance in Bing Webmaster Tools Public Preview", Feb 2026 - https://blogs.bing.com/webmaster/February-2026/Introducing-AI-Performance-in-Bing-Webmaster-Tools-Public-Preview
- Microsoft Bing, "New AI Visibility Insights in Bing Webmaster Tools: Intents, Topics, Citation Share, Compare", Jun 2026 - https://blogs.bing.com/search/June-2026/New-AI-Visibility-Insights-in-Bing-Webmaster-Tools-Intents-Topics-Citation-Share-Compare
