Can an emerging field called ‘neural systems understanding’ explain the brain?

Had you stumbled into a certain New York University auditorium in March 2023, you might have thought you were at pure neuroscience conference. In fact, it was a workshop on artificial intelligence—but your confusion could have been readily forgiven. Speakers talked about “ablation,” a procedure of creating brain lesions, as commonly done in animal model experiments. They mentioned “probing,” like using electrodes to tap into the brain’s signals. They presented linguistic analyses and cited long-standing debates in psychology over nature versus nurture.

Plenty of the hundred or so researchers in attendance probably hadn’t worked with natural brains since dissecting frogs in seventh grade. But their language choices reflected a new milestone for their field: The most advanced AI systems, such as ChatGPT, have come to rival natural brains in size and complexity, and AI researchers are studying them almost as if they were studying a brain in a skull. As part of that, they are drawing on disciplines that traditionally take humans as their sole object of study: psychology, linguistics, philosophy of mind. And in return, their own discoveries have started to carry over to those other fields.

These various disciplines now have such closely aligned goals and methods that they could unite into one field, Grace Lindsay, assistant professor of psychology and data science at New York University, argued at the workshop. She proposed calling this merged science “neural systems understanding.”

“Honestly, it’s neuroscience that would benefit the most, I think,” Lindsay told her colleagues, noting that neuroscience still lacks a general theory of the brain. “The field that I come from, in my opinion, is not delivering. Neuroscience has been around for over 100 years. I really thought that, when people developed artificial neural systems, they could come to us.”

Artificial intelligence, by contrast, has delivered: Starting with visual perception a decade ago and extending to language processing more recently, multi-layered or “deep” artificial neural networks have become the state-of-the-art in brain modeling, at least for reproducing outward behavior. These models aren’t just idealized versions of one aspect of the brain or another. They do the very things natural brains do. You can now have a real conversation with a machine, something impossible a few years ago.

“Unlike the computational models that neuroscientists have been building for a long time, these models perform cognitive tasks and feats of intelligence,” says Nikolaus Kriegeskorte, professor of psychology and neuroscience at Columbia University, who studies vision at the university’s Zuckerman Institute.

That shift has transformed day-in-day-out scientific methodology. As described at the NYU meeting, working with these systems is not altogether unlike doing an experiment in humans or macaque monkeys. Researchers can give them the same stimuli and compare their internal activity more or less directly with data from living brains. They can create do-it-yourself Oliver Sacks-style case studies, such as artificially lesioned machine brains that recognize all four members of the Beatles but can’t tell a drum from a guitar. Direct interventions of this sort, so useful in distinguishing causation from correlation, aren’t possible in a human brain.

These successes are all the more striking considering that the workings of these systems are so un-brain-like. What passes for neurons and synapses are radically dumbed-down versions of the real things, and the training regimen is worlds apart from how children usually learn. That such an alien mechanism can still produce humanlike output indicates to many scientists that the details don’t actually matter. The low-level components, whether they are living cells or logic gates, are shaped into larger structures by the demands of the computational task, just as evolution works with whatever it has to solve the problem at hand.

“As long as architectures are sufficiently decent, representations are affected much more by the data and training process,” says Andrew Lampinen, a cognitive psychologist at Google DeepMind. “It suggests that predicting and comprehending language are bottlenecked through similar computations across a wide range of systems.”

Given this similarity, scientists should begin to explain brain function in terms of network architecture and learning algorithm, rather than fine-grained biological mechanisms, argued Lindsay, Kriegeskorte and a long list of other eminent computational neuroscientists in a 2019 paper. The machine-learning pioneer Yoshua Bengio, professor of computer science at the University of Montreal and a co-author of the paper, puts it this way: “Neuroscience tends to be more descriptive, because that’s easy; you can observe things. But coming up with theories that help explain why, I think, is where machine-learning ways of thinking and theories—more mathematical theories—can help.”

V

isual neuroscientists have led the way in merging AI and neurobiology, as well they might. The first hardware neural network from the mid-1950s was expressly designed to mimic natural visual perception. That said, over the ensuing decades, AI researchers often gave up any pretense of biological verisimilitude, especially in how they trained their networks. By the 2010s, when image-recognition networks began to identify images as well as people (albeit only on narrowly defined tasks), it wasn’t clear how true to life they really were.

To answer that question, neuroscientists at the Massachusetts Institute of Technology soon developed a basic experimental paradigm that those studying other forms of perception and cognition have since adopted. The first step was to show visual stimuli to monkeys and measure their brain responses. Then they trained a range of artificial neural networks on the same stimuli and extracted their “representations”—output encoding the highest abstractions the networks produced. To compare the brain data with this output—typically just a vector of numbers that have no particular biological meaning—called for a “mapping model,” distinct from the artificial neural network itself. Researchers can build such a model by assuming that the simulated and real brains match on some subset of the data, calculating the relation between them, and then testing this relation on another subset of data.

The approach revealed that monkey brains and artificial networks respond to the same visual stimuli in similar ways. “It was the first time we had models that were able to predict neural responses,” says Katharina Dobs, a psychology researcher at Justus Liebig University Giessen. And this congruity was not merely not ad hoc. These systems were designed and trained to recognize images, but nothing required them to do it like a natural brain. “You find this astonishing degree of similarity between activations in the models and activations in the brain, and you know that so did not have to be,” says Nancy Kanwisher, professor of cognitive neuroscience at MIT. “They’re completely different. One is a computer program, and the other is a bunch of biological goo optimized by natural selection, and the fact that they end up with similar solutions to a similar problem is just, to me, astonishing.”

Since then, these networks have changed the scale at which computational neuroscientists do research. Rather than put forward one model of visual perception and defend it, a paper might compare a dozen at once. In 2018, a group of researchers set up the website Brain-Score.org to rank vision models—now more than 200 of them, each representing some intuition for how the visual cortex might work. All perform at a human level on some task, so the rankings represent finer-grained aspects of the models, such as whether they make the same errors our brains make and whether their reaction times vary as ours do. “That gives us a powerful framework for adjudicating among models,” Kriegeskorte says.

These artificial neural networks have opened up new ways to tackle old questions in vision science. Even though the networks are often as opaque as the brain itself, researchers at least have direct access to their artificial neurons—they are just variables in the machine. Lindsay and her colleagues, for example, turned to an artificial network to explore “grandmother neurons,” or the decades-old idea that some brain cells may fire only when you see your grandmother or some other specific person. Lindsay’s team confirmed that an artificial network trained on images had neurons that fired only when certain objects were present. But when they traced the information flow through the network, they found that these neurons had nothing to do with the network’s overall ability to identify people or things; it was just an accident that they responded so selectively.

“Looking in these artificial neural-network models, we actually see that the way a neuron responds to an image doesn’t necessarily tell you the role it plays in classifying things,” Lindsay says.

A

rtificial neural networks have also made it possible to probe deeper into the hierarchy of visual processing experimentally. Textbook neuroscience methods measure the response of neurons in early processing layers such as the retina, the lateral geniculate nucleus and the primary visual cortex. They show that neurons respond to simple stimuli such as lines and gratings of specific orientations. But these methods struggle to characterize cells in late layers that pick up broader and more complex geometric patterns. “As you go deeper into the visual system, it’s harder to find some simple feature like that that explains how the neurons are responding,” Lindsay says.