Can AI Actually Do Science?
In a recent article, I looked at how, and indeed whether, scientists and researchers were using AI to support their work. Of course, in the usual automation vs augmentation debate, it’s perhaps also worth. . .
In a recent article, I looked at how, and indeed whether, scientists and researchers were using AI to support their work. Of course, in the usual automation vs augmentation debate, it’s perhaps also worth considering whether AI can also play a role in doing science rather than just supporting science. A recentstudy(from more schools than you can shake a stick at) suggests that AI is currently exceptional at passing exams, but is pretty poor at doing what we might regard as real science. Growing expectationsIn the pantheon of artificial-intelligence achievements, 2024 was a banner year. Large language models (LLMs) crushed coding challenges, aced graduate-level mathematics, and demonstrated reasoning abilities that seemed to herald a new era of machine intelligence. Yet the new study suggests that when it comes to genuine scientific discovery, even the most sophisticated AI systems remain surprisingly mediocre. The study, conducted by an international consortium of researchers spanning biology, chemistry, materials science, and physics, introduces what they call the Scientific Discovery Evaluation (SDE) framework. Unlike conventional benchmarks that test whether models can answer decontextualized quiz questions, SDE evaluates whether AI can actuallydoscience. The distinction matters enormously. The researchers started by identifying concrete research projects of genuine interest to domain experts, then decomposed each into modular “research scenarios”—recurring reasoning patterns that scientists encounter in real laboratory and computational work. From these scenarios, they constructed 1,125 expert-vetted questions spanning everything from predicting enzyme reactions to interpreting nuclear magnetic resonance spectra. Crucially, every question connects directly to an authentic research challenge, ensuring that correctness reflects practical progress rather than trivia mastery. Lack of crossoverThe results are sobering. State-of-the-art models from OpenAI, Anthropic, and other leading developers achieve scores of 60-75% across scientific domains. This is a respectable showing, but markedly worse than their 84-86% performance on conventional science benchmarks, such as GPQA-Diamond. The gap reveals a fundamental truth: excelling at standardized tests does not guarantee readiness for discovery, just as earning perfect grades does not indicate research brilliance. More troubling still, the gains from two strategies that have driven recent AI breakthroughs—scaling up model size and increasing test-time reasoning—show diminishing returns. Between medium and high reasoning efforts, GPT-5’s accuracy improves negligibly (from 0. 70 to 0. 69 in biology, 0. 74 to 0. 75 in materials). Even as these models set new records on coding and mathematics benchmarks, their scientific discovery capabilities plateau. Perhaps most revealing is what happens when top performers from different companies are compared. The leading models from OpenAI, Anthropic, xAI, and DeepSeek exhibit remarkably correlated error patterns, frequently converging on identical wrong answers. This suggests they share common systematic weaknesses, likely inherited from similar training data rather than from architectural differences. In practical terms, this means that simply combining predictions from multiple providers delivers limited improvement on the hardest scientific questions. When knowledge gaps don’t matterThe framework’s most striking innovation lies in its two-tier evaluation structure. The researchers wanted to test more than just competence on individual questions, they also wanted to gauge whether AI could drive research projects, propose new hypotheses, execute simulations, and then refine ideas. This “project-level” evaluation reveals capability gaps that static quizzes cannot capture. The results defy simple patterns. Models excel at projects involving well-structured data and codified knowledge, whether that’s protein design, materials optimisation, or symbolic regression, yet struggle with endeavors requiring rigorous long-term planning, such as multi-step chemical synthesis. Surprisingly, strong performance on constituent scenarios does not always translate to project success. LLMs achieve excellent results when optimizing transition metal complexes despite scoring poorly on questions about their properties, suggesting that discerning optimization directions matters more than knowing explicit structure-property relationships. Conversely, models that score highly on retrosynthesis questions fail to generate valid multi-step synthesis routes, frequently violating basic validity checks. The capacity for serendipitous exploration, it seems, can compensate for imperfect granular knowledge, whilst detailed understanding provides no guarantee of effective discovery. The road aheadWhat explains AI’s scientific shortcomings? There are a number of possible factors. For instance, the way we train models today prioritizes breadth over depth in scientific reasoning. Models also find it difficult to integrate domain-specific tools that are commonly used by researchers each day. The researchers also found that the kind of reinforcement learning-based strategies that work well in areas like coding and mathematics have much less impact on other scientific tasks. This reminds us that we need different approaches for things like scientific discovery. These targeted interventions are key to greater success with AI in science. We need to shift our focus away from indiscriminate scaling and towards a focus on generating and testing hypotheses. A greater diversification of pre-training data might help, not least as it would mitigate the shared failure modes that plague current providers. An uncomfortable truthThe study delivers an uncomfortable message to those prophesying imminent scientific superintelligence: no current model wins across all projects evaluated. Leadership rotates depending on the task. All systems remain distant from true general scientific capability. The machines that dazzle with their examination prowess still struggle with the patient, iterative work that drives actual discovery. That need not cause undue pessimism, of course. LLMs already demonstrate promise across diverse scientific challenges, sometimes succeeding even where their constituent knowledge appears weak. With human oversight, they accelerate literature review, hypothesis generation, and computational simulation. The question is not whether AI will transform scientific research, but whether it will transcend its current role as a capable assistant to become an autonomous discoverer. The evidence suggests that the journey will be longer and harder than the hype implies.