Artificial Intelligence 2024: Smarter But More Expensive

After a year of fast and furious development of generative AI, the industry is at a crossroads. The technology is poised to deliver unprecedented productivity growth but may be hamstrung by limits to the technology as well as guardrails for its usage.

That’s the word from Stanford University’s Institute for Human-Centered AI (HAI), which just released the 2024 edition of its annual AI Index. Interestingly, while AI continues to become mainstream, the study’s authors observe that the investment wave and the need for AI development skills has waned.

There appear to be some doses of reality coursing through the business world — AI is a powerful tool, but it’s nowhere near the point at which it can seamlessly take on the bulk of work.

“AI beats humans on some tasks, but not on all,” the report’s authors state. “AI has “surpassed human performance on several benchmarks, including some in image classification, visual reasoning, and English understanding. Yet it trails behind on more complex tasks like competition-level mathematics, visual commonsense reasoning, and planning.”

There’s no question that AI has become smarter and more powerful over the past 12 months. At the same time, the costs of building and maintaining large language models (LLMs) has increased astronomically, In addition, the industry still lacks standards for responsible AI best practices.

The number of new large language models released worldwide in 2023 doubled over the previous year, the report states. “Two-thirds were open-source, but the highest-performing models came from industry players with closed systems.

Gemini Ultra became the first LLM to reach human-level performance on the Massive Multitask Language Understanding (MMLU) benchmark; and performance on the benchmark has improved by 15 percentage points since last year.

LLMs have also grown way more expensive, the HAI authors also observe. “For example, OpenAI’s GPT-4 used an estimated $78 million worth of compute to train, while Google’s Gemini Ultra costs $191 million for compute,” they estimate.

At the same time, generative AI investment skyrocketed over the past 12 months. “Funding for generative AI surged by a factor of eight, from 2022 to reach $25.2 billion. Major players in the generative AI space, including OpenAI, Anthropic, Hugging Face, and Inflection, reported substantial fundraising rounds.”

Those working to design, build, and implement AI systems need to be more open about their methods, the report also suggests. “AI developers score low on transparency,” the co-authors suggest. “This is especially the case “regarding the disclosure of training data and methodologies. This lack of openness hinders efforts to further understand the robustness and safety of AI systems.”

Responsible AI is still an open and incomplete effort. “Robust and standardized evaluations for LLM responsibility are seriously lacking,” the HAI authors report. There is “a significant lack of standardization in responsible AI reporting. Leading developers, including OpenAI, Google, and Anthropic, primarily test their models against different responsible AI benchmarks. This practice complicates efforts to systematically compare the risks and limitations of top AI models.”

It’s no surprise, then, that the number of AI regulations in the United States sharply increased. In 2023, there were 25 AI-related regulations, up from just one in 2016. In the past year, the total number of AI-related regulations grew by 56%. Regulation have come out of the U.S. Department of Transportation, Department of Energy, and the Occupational Safety and Health Administration.

Another issue surfacing over the past 12 months is intellectual property and copyright violations, as generative AI synthesizes existing information from many sources. “Multiple researchers have shown that the generative outputs of popular LLMs may contain copyrighted material, such as excerpts from The New York Times or scenes from movies,” the HAI researchers point out. “Whether such output constitutes copyright violations is becoming a central legal question.”