What’s the difference between an AI Overview, a Gemini Pro answer and a Gemini Pro answer when you force a web search?

Do your chats actually retrain the core training knowledge? How long does it take for the core ‘encyclopaedia’ to get updated and what are the triggers?

What does this mean for users looking for answers and what does this mean for marketers wanting to be cited in answers?

I love an LLM rabbit hole.

This one was triggered by the caveat at the bottom of my Gemini Pro chat window. It read, “Your algorhythm.london chats aren’t used to improve our models.”

I instantly thought about clients and colleagues who spend hours trying to ‘retrain’ the LLMs because they have been served answers with out-of-date or incorrect data about their products and brands.

I thought to myself, Huh? What do you mean my chats do not improve your model?

What followed was a conversation with Gemini that was fascinating and a little scary – and I mean that as an ‘all-in’ adopter of AI and especially LLMs, as someone seeking answers but also as an agency SEO specialist trying to unravel the impact of LLMs on search.

Here’s a couple of assumptions I’d made, that turned out to be completely false:

Gemini Pro, as a paid premium service, will be the most accurate source when looking for answers. WRONG!

AI Overviews are simply ‘Gemini Light’ answers, and whilst important for zero click search, can’t be trusted as much as an actual chat window. WRONG!

All of our combined chats will be training the LLM and will improve the accuracy of information over time. WRONG!

Let’s unpick those assumptions.

Gemini Pro, as a paid premium service, will be the most accurate source when looking for answers.

Gemini Pro relies on its core training knowledge, like a giant, out of date encyclopaedia. It is brilliant for scientific or technical tasks, but it’s like asking a scientist locked in a windowless room, to tell you about the weather.

If you force Gemini Pro to do a web search, it will do a surface level search but does not have access to the full web index. Although I have trained Gemini in our chat, it will forget the conversation as soon as I leave that window. It has not trained (updated) the core training knowledge in any shape or form.

If I ask Gemini Pro to compare one of my clients with its main competitors, it suggests it has 30 products. In reality, my client has 40 products and that is very clear on their website. If I point this out to Gemini, I get the obligatory apology and a more up-to-date comparison.

The next time I ask for a comparison, it will state that the client has 30 products. If the rest of the world asks, it will state the client has 30 products. That may get updated in the next core update (3-6 months) or it could be in 12 months or it could be in 18 months. Who knows?

AI Overviews are simply ‘Gemini Light’ answers, and whilst important for zero click search, can’t be trusted as much as an actual chat window.

AI Overviews are considered the Gold Standard in terms of AI answers. AI Overviews have access to the entire web index and get a real-time answer for users as well as access to the core training knowledge. If your website, schema and other signals are correct, the AI Overview is likely to be bang on.

All of our combined chats will be training the LLM and will improve the accuracy of information over time.

Unfortunately, any time you spend ‘training’ Gemini Pro is wasted time. It will only improve the accuracy of your chat in that moment – it certainly doesn’t send a signal to update the core knowledge. Like a massive ship, the core encyclopaedia is slow and takes an age to update. Every time someone clicks on a thumbs down, that goes into a giant database and if enough (thousands) of users have clicked on a thumbs down, an engineer ‘MIGHT’ flag a need for an update but even here, it’s mainly about systems and the way the AI is thinking or constructing its answer – much less likely to actually update the information.

The core training will do live web scans to periodically (every 6 months) update its information and it will use websites, schema, reviews, socials, Reddit to update product and brand information.

Here’s a cheeky FAQ to massage the AI Overview: