AI answers shifted 84% overnight—even the same questions got new brands

We asked AI the same 210 buying questions 2 days in a row,84% of answers changed

Supercite asked 210 buying questions across ChatGPT, Gemini, Google AI Overviews and AI Mode on two consecutive days. 84% of answers named a different set of brands the next day, 27% of brands vanished entirely, and only 42% of cited pages stayed the same. Even a #1 pick was still #1 just 63% of the time. Smaller brands kept only half their spots versus 73% for category leaders.

An AI answer isn't a fixed ranking. It's a summary of a few pages, picked fresh each time.
  1. ryanisnan

    I don't think this is inherently surprising.

    Looking at the questions they asked, they are quite broad. In my experience, part of the value of LLMs comes from its randomness, but you must understand it as a characteristic virtue of these tools, and must adapt accordingly.

    Asking questions with tight scopes, clear constraints, and specific outcomes, always yields better results, in my opinion.

  2. mentalgear

    People think "AI" has all the facts to make objective decisions, but it seems to boil down to (SEO again?) what pages its search engine returned that day.

    > If you're not a leader, this is the part that matters. Your visibility depends on which pages the engine reads that day. The way to make it stable is to be on more of those pages.

    Ultimately a true facts knowledge graph independent of mentions is needed.

  3. hnlmorg

    All you’re proving is that any specific domain will have a dozen mature solutions.

    What separates one product from another is going to be constraints specific to the project you’re integrating this into. And none of that is defined in your prompts. So you’re effectively just asking an LLM to roll the dice and pick something at random and then acting surprised means a different answer each time.

    Edit: their conclusions at the end of the article miss the most important point. They talk about asking multiple times over multiple days, but that’s just repeating the “garbage in -> garbage out” cycle without addressing the root cause of the problem: not defining your requirements properly.

    And if you don’t have any requirements beyond “something cheap” (for example) then who cares what answer you get when any result is equally valid for your use case.

  4. seabre

    If LLM output is stochastic, why would you expect it not to change with questions that don't have settled answers?

  5. luisfominaya

    In general it seems like LLMs have a fundamental problemw with noise in their outputs such that randomness (what data they pull and how they aggregate it) has an outsized impact when asking it for opinions/recommendations/forecasts.

More from this day

2026-11-18