Why Measuring the ROI of Transformative GenAI is So Hard
Google’s market cap topped $2 trillion last week after quarterly earnings convinced investors that demand for its GenAI platform was not only strong but being monetized in the here and now. Microsoft, according to reports, attributed 7 points of its 31% quarterly cloud revenue jump to GenAI use cases. The CFOs of both whispered the possibility of the near-term risk of demand outpacing supply, given the almost universal acceptance of the potential for GenAI’s transformational impact on people, business, and society — and the fear of being left behind.
We see this too. A recent PYMNTS Intelligence study of how enterprise CFOs view GenAI’s impact on their business finds that nearly two-thirds believe it is the most significant technological innovation of our time. They use words like “growth,” “success,” “impact,” “innovation,” and “future” when asked to provide a one-word description. None of the CFOs surveyed expressed a negative view, even though some admitted the need to know more about its application to their business.
Although the study finds investments in GenAI to be relatively modest and experimental today, 53% of CFOs say that they’ll loosen the purse strings and significantly increase those budgets company-wide in the next year. Even the CFOs who acknowledge its potential, but have been more conservative, are ready to jump in. Nearly two-thirds (65%) of those CFOs expect to increase their corporate-wide GenAI budgets by an average of 12% in the coming year.
The Need for a GenAI Organizing Framework
What’s missing in all the coverage of the models, investments, startups and GenAI applications is a framework for analyzing the link between the application of the technology and business outcomes — and how executives across an organization measure the impact of their investments to support it.
Looking at GenAI through that lens can inform the characteristics of the companies and industry segments that recognize it as a strategic enabler to their business early on. More important, it offers a consistent way to track the impact of GenAI on business and financial performance over time. As GenAI and LLM applications become an embedded part of a business, the only way to measure the impact of this technology will be by looking at its impact on overall business outcomes.
That is the thesis of a new monthly PYMNTS Intelligence study we call The CAIO™ Study. The first of these monthly reports will be released on Tuesday, May 14th.
The overall study framework consists of a proprietary weighting and scoring methodology for each of 13 activities to classify them as routine or strategic uses of GenAI. The PYMNTS Intelligence team then uses statistical techniques to examine how each activity, and groups of activities, can influence business outcomes and the return on their investments to support it, as reported by the C-suite. Each month will study a different enterprise C-Suite executive and in order to do a deeper dive into a particular business function.
The study branding is a nod to the speed at which GenAI is becoming an embedded part of business decision-making and executive proficiency. Successful businesses will be led by executives who see GenAI as the invisible engine that powers the business and who prioritize the people, process, and dollars to make it real for themselves and their customers over the next one, three and five years.
The CAIO™ Study will methodically benchmark that journey across the enterprise C-Suite and expects to have data on more than 700 such U.S. executives in its first year.
The May 14th release will share what we learned after conducting a double-blind survey of a random sample of CFOs of 60 of the largest companies in the U.S. — $1 billion or more in annual revenue. For this issue of the study, we drilled into the applications of GenAI across the business, what is being invested to support those activities, and how CFOs currently measure its impact.
Starting with the fact that not even enterprise CFOs who’ve invested the most have a clear vision for how to effectively predict the impact of GenAI on the financial performance of the company in these early days.
A Whole Lot of Writing Emails Going On
Here’s the good news: Right now, nearly all enterprise firms we studied are using GenAI to some degree.
Here’s the highly predictable news: Most of those CFOs have yet to see a demonstrable return from those investments.
There are a few very good reasons for that.
First, the enterprise firms studied are just getting their feet wet with mostly low-risk, low-impact, less complex applications of GenAI tech.
And second, those types of applications and use cases represent the bulk of their company’s experiences to date with GenAI. Most of what we see so far inside of these enterprise firms is the use of GenAI to create emails, summarize reports, search for information, produce graphics and write social media posts.
I won’t ask whether the last email you sent also had a little GenAI helping hand.
Roughly half of the GenAI use cases inside these firms can be classified this way — more routine than strategic, more for internal use than customer-facing, and mostly by individuals inside of the company to save time and improve the quality of their personal workflows.
It’s why we also find a relationship between the number of firms using GenAI to tackle more complex activities and the impact CFOs report GenAI has on the business right now.
Only 14% of the CFOs in our study say that GenAI-ing routine, less complex activities deliver a positive impact on the business. It is hard to measure the collective financial and business impact of better emails and snappier reports.
They have a much different view when LLMs are used in broader and more strategic ways.
More than twice as many CFOs report a strong ROI when GenAI is applied this way, even as CFOs right now are using more gut instinct than P&L performance to judge its potential impact. Strategic activities in this study include innovating products and improving processes and workflows for mission-critical business functions such as the production process, fraud and cybersecurity.
We also observe a surprising disconnect between what conventional wisdom suggests as popular strategic applications of GenAI and how CFOs project their bottom-line impact today.
For example, chatbots are widely popular and often talked about as a GenAI use case; it’s also one that CFOs say is an effective application of the tech. Yet we find roughly a third of enterprise CFOs (32%) report a negligible impact of chatbots on financial and business performance. It could be that the tech is still new and being introduced alongside people sitting in call centers, making it hard to predict its impact on business bottom line right now.
According to the study, using GenAI to generate code is a less widely used case, but one that CFOs report as having 2.5 times greater impact on business performance than others. CFOs may find it easier to conceptualize GenAI’s impact on the cost of supporting a development team, for example, when coding has the potential to become more widely distributed across the business.
More GenAI Means More Impact
Nearly every firm in The CAIO™ Study uses GenAI to support an average of four routine, less complex, non-mission-critical tasks, as described earlier. Enterprise firms that use GenAI more strategically use an average of six applications across the business.
We also find that enterprise CFOs using six or more GenAI applications report an eightfold increase in the expected ROI impact of GenAI on the business compared to those using five or fewer. One might expect that going from four to six applications would be linked to CFOs reporting a change in ROI impact proportional to that increase. But we see that the addition of a few strategic applications can be quite impactful.
That’s even as only one in three CFOs studied are using the technology across the business to this degree (i.e. six or more applications).
We also find that when CFOs have more experience using GenAI applications to support strategic business initiatives, they have a greater appetite to invest more in its strategic use. Those CFOs cite an average investment in GenAI applications at $4.6M, 88% more than those whose use cases are more routine.
What Gets Managed Gets Measured, Sort Of
Nearly a year and a half after the public launch of OpenAI and Chat GPT, we find that the methods enterprise CFOs use to measure the future value of their current investments in GenAI applications are still a work in progress.
It is admittedly early days, and one study of a small sample of enterprise CFOs does not make a trend. But we believe that what CFOs report as the drivers of a positive ROI may also undervalue GenAI’s potential business impact in the years to come. There’s no data to benchmark or use as a guide right now. It is also difficult to forecast the impact across the business of such a transformative technology.
We already see signs of how measuring GenAI’s impact might take shape. The enterprise CFOs who report that their firms have embedded GenAI into more complex, higher-value business functions also say the business and financial impact is stronger, and their reports of a positive ROI are more certain.
They report a more positive impact across the business in the areas of customer delivery, critical business process improvement and competitive positioning. More than three quarters (78%) of the firms who use GenAI this way count positive impacts in new product innovation, and 72% see positive impacts on market adaptability and business expansion — nearly twice as much as those who haven’t expanded their GenAI wings across the business.
Some of that can be explained by how CFOs see the staffing mix evolve over time and how it impacts the costs to support the workforce in the here and now.
Eight in ten enterprise CFOs in this study say that GenAI will have a big impact on staff mix, with nearly two thirds saying that their need for lower-skilled workers across the business has decreased at the same time the need for more analytically skilled workers has increased. CFOs have a direct line of sight into those staffing plans and forecasts, so they may be using it as a proxy for measuring business performance over the longer term.
We also see evidence that it is the blend of GenAI applications across both routine and strategic use cases that drives a stronger, more positive ROI outlook as reported by the enterprise CFOs we studied
And why we find little evidence that any single GenAI application is the “silver bullet” to take the overall business performance of the largest companies in the U.S. to the next level.
