Google AI Overviews search tool stumbles. It won’t be the last

When Google unveiled its AI-powered search feature last month, it boasted that the new tool would “do the Googling for you.” Called AI Overviews, the new capability serves up concise, AI-generated summaries with key info and links, displayed atop search results.

The internet, as it does, quickly humbled the tech giant, checking its capabilities like the raptors in Jurassic Park testing the fences for weakness. People found AI Overviews giving some unhinged answers to searches, including suggestions like eating one rock a day and using glue on pizza.

Liz Reid, head of Google Search, wrote a blog post last Thursday trying to explain it all. Many of the searches were so uncommon that there wasn’t any good information to pull from, something she called a “data void.” Other causes, including the one rock a day example, were pulled from satirical sites, like The Onion. In the mea culpa, she made sure to note that it wasn’t quite as bad as it looked on social media, writing, “[T]here have been a large number of faked screenshots shared widely.”

AI Overviews got so much attention because it was a new use case for generative artificial intelligence that could expose anyone who has missed the AI craze to the technology by bringing it front and center to an internet activity that many people do a dozen times a day. As big and as quickly as generative AI has gotten — ChatGPT reached 100 million active users in two months, blowing away TikTok’s previous record of nine months — Google’s reach could dwarf that.

There are 8.5 billion Google searches per day — at least according to an AI Overviews answer to the question, “How many Google searches are there a day?” AI Overviews is an opportunity for the tech giant to show a lot more people the power — and peril — of generative AI, one Google has a lot riding on.

Reid promised fixes, but more issues are going to surface as the large language models (LLMs) that power AI chatbots get rolled into more products, said Delip Rao, an independent researcher associated with the University of Pennsylvania. In the case of AI Overviews, Google ran into two main problems, Rao said: bad data and “hallucinations.”

The internet is filled with bad data. Google can limit the sites it pulls from and put up more guardrails for sensitive searches, like those about health topics, both of which Reid said the company plans to do. But hallucinations — what it’s called when generative AI seems to make up information — are a problem without an easy solution, Rao said.

“Whatever makes the model creative, also makes it responsible for hallucinations,” he said. “You can’t have one without the other.”

Rao, who has used AI to build a fact-checking tool, still has hope that fixes like automated fact-checks, controversy detection, and credibility scores could improve AI Overviews.

“AI in search won’t go away,” he said. “We will be stuck with this. [But] a lot of the problems will be whittled down.”

In a statement, Google said it is already implementing improvements, saying it has made more than a dozen technical updates to AI Overviews.

“​​AI Overviews are helping people on a large number of queries on Search today, serving as a jumping off point to content across the web,” a Google spokesperson said.

AI all the way down

Tech industry watchers have seen Google parent Alphabet as playing catchup since OpenAI debuted ChatGPT in November 2022. Google has made progress — revenue was up 15% in its latest earnings report in April, and its chatbot, which uses the same technology that powers AI Overviews, ranked third in a recent Wall Street Journal test of chatbots.

But AI Overviews isn’t Google’s first AI misstep. The company took down its AI image generator after the tool created historically inaccurate images, including pictures of racially diverse Nazis.

Despite the mistakes, Olivier Toubia, a Columbia University marketing professor, said Google has taken a more prudent approach so far compared to other companies building AI models. That was probably intentional, he said.

“Sure maybe they lose this cutting edge about being at the forefront, but at the same time, they might build the perception that they are the responsible tech firm,” Toubia said.

But a cautious approach eventually has to give way to market needs. Google launched AI Overviews after the AI search startup Perplexity and Microsoft’s Copilot started bringing AI to search in a way that could threaten Google’s dominance.

The search giant probably has an advantage, with people more likely to use one familiar company instead of two for their generative AI and search needs. Convenience may make users more forgiving. Consumers have long since been conditioned not to believe everything on the internet — so being suspicious of an AI-powered Google search wouldn’t be new.

“People aren’t going to walk away from this and say, ‘I’ll never use generative AI again,’” Toubia said.

But as Google adds more AI functionality to other products, issues could crop up that people might be less willing to forgive — such as bias and privacy problems.

“There are more concerning things than being told to use glue on pizza,” Toubia said.