Payoff from AI projects is ‘dismal’, biz leaders complain • The Register

Businesses have become more cautious about investing in artificial intelligence tools due to concerns about cost, data security, and safety, according to a study conducted by Lucidworks, a provider of e-commerce search and customer service applications.

“The honeymoon phase of generative AI is over,” the company said in its 2024 Generative AI Global Benchmark Study, released on Tuesday. “While leaders remain enthusiastic about its potential to transform businesses, the initial euphoria has given way to a more measured approach.”

Between April and May 2024, Lucidworks conducted a survey of business leaders involved in AI adoption in North America, EMEA, and the APAC region. The respondents, it’s claimed, were drawn from 1,000 companies with 100 or more employees across 14 industries, all of which are said to have active AI initiatives underway.

About 23 percent are executives and about 50 percent are managers, with 86 percent involved in technology decision making. Thirty-nine percent of participants hailed from North America, with 36 percent from EMEA, and 24 percent from the APAC region.

According to the results of the survey, 63 percent of global companies plan to increase spending on AI in the next twelve months, compared to 93 percent in 2023 when Lucidworks conducted its first investigation.

Broken down by region, the highest percentage of those planning to spend more was documented among US respondents, at 69 percent. But in the APAC region, less than half (49 percent) of Chinese business leaders expect to increase AI spending, which is down from last year when every one polled said they would be spending more.

Among all organizations, 36 percent planned to keep AI spending flat in 2024, compared to just 6 percent last year.

The slowdown can be attributed to several factors.

One issue is that AI has not yet paid off for those trying to make it work. “Unfortunately, the financial benefits of implemented projects have been dismal,” the study says. “Forty-two percent of companies have yet to see a significant benefit from their generative AI initiatives.”

And to realize meaningful benefits, it’s necessary to move past the pilot testing stage, something few companies have managed. Only 25 percent of planned generative AI investments have been completed to date, the study says.

Another reason is growing concern about the cost of AI projects, up 14x since last year. There are also many more worries (5x more) about the accuracy of responses provided by AI systems.

The costs involved in using generative AI strategically can really add up

With regard to cost, some 49 percent of organizations have opted to use commercial LLMs like Google’s Gemini and OpenAI’s ChatGPT. Another 30 percent use both commercial and open-source LLMs, while just 21 percent have bet exclusively on open-source LLMs like Llama 3 and Mistral. The Lucidworks study predicts the balance will shift toward open-source LLMs based on anticipated open-source model performance gains and on cost considerations.

“Absolutely the costs involved in using generative AI strategically can really add up, regardless of whether you’re hosting your own large language models or using commercial APIs,” said Eric Redman, senior director of product for data science at Lucidworks, in an email to The Register.

“These initial costs are comparable in the grand scheme of things, but they’re really just the tip of the iceberg.”

According to Redman, security and accuracy, response alignment with policies, data acquisition costs, and keeping costs under control all need to be considered.

“The bottom line is that ensuring AI security, accurate AI responses, and responsible data acquisition all come with a price tag,” said Redman. “Cutting corners in these areas can lead to inaccurate or inappropriate responses, which ultimately undermines the value and effectiveness of your AI implementation.”

Among the organizations surveyed, the best generative AI initiatives involved governance (standardizing models to ensure alignment, limiting access to generative AI tools based on role, and so on) and cost reduction, both general and administrative (Q&A testing, debugging, code suggestion, and HR help documentation).

The survey indicates that qualitative applications, which use text and provide a narrow response, have been the most successful generative AI initiatives, accounting for about a quarter of successful implementations. Specifically, these include projects for generating FAQs and providing HR support. And they tend to be the simplest to implement.

Applications with a quantitative component – using generative AI to monitor, predict, analyze, optimize, prioritize, and other more challenging projects – have had a rougher time, with less than 15 percent successfully implemented. As the study observes, these involve optimizing search results, screening job applicants, supporting financial result closure, and so on.

Redman said code generation’s popularity as a top use case makes a lot of sense. “It’s essentially a prime example of how AI copilots can empower knowledge workers,” he explained.

“These copilots have proven to be valuable across various creative tasks, whether it’s writing code, drafting documents, or beyond. The beauty lies in their collaborative nature – they offer suggestions and support, but the final decision and responsibility for the output rests firmly with the human user. This is a stark contrast to, say, a chatbot interacting with customers, where the AI might have a greater degree of autonomy.”

And Redman said it’s also not surprising that AI governance has been a common key initiative.

“In the face of powerful generative AI capabilities, organizations are naturally prioritizing risk management,” he said.

“Understanding and mitigating the risks associated with each AI application is paramount, especially considering the growing regulatory landscape, like the EU AI Act. These regulations emphasize transparency and user control, ensuring individuals understand how their data is used and have choices in their interactions with AI systems.” ®