BrainBench: AI Prediction Model Uses LLM Technology to Outperform Neuroscientists by 23%

Upon evaluating the predictive abilities of large language models (LLMs), University College London (UCL) researchers found that the AI models could predict the results of proposed neuroscience studies with greater accuracy than human experts. The research points to the potential of AI to accelerate scientific progress when applied in neuroscience.

The research results, published in Nature Human Behaviour, indicated that LLMs outperformed human neuroscientists consistently in predicting neuroscience study outcomes, averaging 81.4 percent accuracy compared to 63.4 percent for human experts.

Measuring AI Prediction Accuracy

To evaluate the ability of LLMs to predict study results, UCL researchers developed a benchmark for neuroscience called BrainBench. The tool quantified and compared the predictive ability of 171 human neuroscience experts and 15 general-purpose LLMs. Researchers tested their abilities to identify the genuine abstract between two pairs of study abstracts, containing one real abstract and one with modified results.

The study’s outcomes revealed that the accuracy of human experts continuously fell short compared to that of the AI models. Although all humans participating in the study passed a screening test to predict neuroscience research outcomes, even those with the most expertise in specific neuroscience domains showed less accuracy (66 percent) than the LLMs.

The research involved test cases spanning each area of neuroscience, including cellular/molecular, behavioral/cognitive, systems/circuits, development/plasticity/repair, and neurobiology of disease. The LLMs evaluated included different versions of Falcon, Llama, Mistral, and Galactica. In response to these findings, the researchers developed BrainGPT, a new LLM created using an existing version of Mistral trained specifically on neuroscience literature amassed from journals published from 2002 to 2022. The BrainGPT model demonstrated an even more advanced level of accuracy, at 86 percent.

The Future of AI in Neuroscience

The study’s findings indicate the potential of predictive artificial intelligence technology to enhance neuroscience processes and accelerate scientific progress.

“Scientific progress often relies on trial and error, but each meticulous experiment demands time and resources,” said Ken Luo, the UCL study’s lead author. Even the most skilled researchers may overlook critical insights from the literature. “Our work investigates whether LLMs can identify patterns across vast scientific texts and forecast outcomes of experiments.”

The study’s senior author, Professor Bradley Love agreed. “This success suggests that a great deal of science is not truly novel, but conforms to existing patterns of results in the literature,” he said.

In the future, incorporating strategically trained LLMs within the larger systems neuroscientists use in their research could help them to determine which experiments to conduct and guide them to make optimal decisions regarding limited resources like money and time, enhancing research efficiency.

The post BrainBench: AI Prediction Model Uses LLM Technology to Outperform Neuroscientists by 23% appeared first on eWEEK.