Note, August 2026: after recent advances in AI for mathematics, my views on some of what follows have changed. I intend to write and post an updated document soon. As of now, I am a spectator.
Artificial intelligence is increasingly identified with large language models (currently based on the transformer architecture [1]) systems capable of producing fluent text, writing software, and answering questions with remarkable ease. Yet, this rapid and evident progress risks obscuring a more fundamental issue: What precisely do contemporary AI systems learn, and how does that learning relate to scientific understanding?
The recent surge in generative models has revived long-standing questions about the nature of intelligence, learning, and explanation. Although performance has advanced at an extraordinary pace, conceptual clarity has not kept up. As AI becomes a staple in scientific research, medicine, and decision-making, it is increasingly important to distinguish predictive capability from understanding, and automation from insight.
Learning at Scale
Modern AI research is dominated by data-driven learning systems, particularly deep neural networks trained at unprecedented scale. These systems differ from earlier approaches not because they implement fundamentally new ideas, but because they exploit vast datasets and computational resources to learn internal representations directly from raw inputs. This emphasis on representation learning rather than explicit rules or symbolic reasoning has proven enormously effective.
End-to-end trained neural networks can extract structure from images, speech, and text with minimal prior assumptions, without suffering from overfitting. Classical statistics, however, was unable to explain this phenomenon, known as Double descent. Yet this success has come at a cost. As models grow, their internal mechanisms become harder to characterize, their training dynamics more opaque, and their behavior more difficult to predict outside carefully curated benchmarks. The field has advanced empirically, but with limited theoretical consolidation.
Nobel laureate Geoffrey Hinton has argued that while AI holds enormous promise, its rapid progress poses serious risks if not matched by careful regulation and human oversight [2].
“The Bitter Lesson”
This trajectory reflects a broader pattern identified by Richard Sutton (considered the father of RL and Turing Award winner 2024), who argued that the largest gains in AI have historically come not from human-designed structures, but from methods that scale with computation and data. In reinforcement learning, agents improve performance by interacting with an environment and optimizing reward through repeated trial and error.
Over time, approaches that rely on general-purpose learning and massive experience have consistently outperformed more carefully engineered alternatives. This observation by Richard Sutton, often referred to as “the bitter lesson” [3], is uncomfortable. It suggests that progress in AI owes less to insight into intelligence and more to brute-force optimization. From this perspective, intelligence emerges not from elegant theory, but from relentless statistical pressure applied at scale. The lesson is not that theory is useless, but that it has played a surprisingly small role in recent breakthroughs.
Scale Versus Structure
Contemporary AI research is heavily invested in scaling laws: empirical regularities that relate model size, dataset size, and performance. These laws have been remarkably predictive and have guided the construction of increasingly large systems. However, scaling alone offers limited guidance about how models work, why they fail, or how they might generalize beyond their training regimes.
In response, there is renewed interest in embedding structure and prior knowledge into learning systems. Physics-informed neural networks, for example, incorporate conservation laws or differential equations directly into model architectures or loss functions. Similar ideas appear in causal modeling, equivariant networks, and hybrid symbolic-neural systems. These approaches reflect a growing recognition that inductive bias matters not only for efficiency, but for interpretability and reliability. Pure scale may deliver performance, but structure may be essential for scientific understanding.
AI as a Scientific Instrument
AI is already reshaping scientific practice. Perhaps the most prominent example is protein structure prediction, where deep learning systems have achieved accuracy that rivals experimental methods. The Nobel Prize in 2024 for chemistry was awarded for the same [4]. These advances have accelerated research in biology and medicine, enabling rapid hypothesis generation and guiding experimental design.
At the same time, these successes highlight an important asymmetry. AI systems can predict outcomes with high accuracy while offering limited insight into underlying mechanisms. In protein folding, models can infer structure from sequence without explicitly representing the physical processes that produce it. The result is a powerful predictive tool, but not a complete explanatory theory.
As emphasized by Terence Tao (during his talk at IMO 2024), current AI systems already assist mathematicians in drafting arguments, exploring proof strategies, and formal verification. Yet their role remains fundamentally supportive: they accelerate exploration without supplying the conceptual insight that guides it. AI can suggest patterns and fill gaps, but deciding what is worth proving and why remains a human responsibility [5].
The Problem of Interpretability
One of the central tensions in modern AI is interpretability and explainability. A range of techniques, such as feature attribution, saliency maps, and dimensionality reduction, aim to explain model predictions by identifying influential inputs or internal components. These methods are often useful in practice, particularly in applied settings where transparency is required. However, such explanations are typically post hoc and qualitative. They describe correlations within trained models rather than causal mechanisms.
Prediction Without Explanation
Science seeks explanations: why phenomena occur and how they arise from underlying principles. By contrast, many AI systems excel at producing answers without providing reasons. For instance, data-driven weather models such as GraphCast and Pangu-Weather have surpassed traditional physics-based dynamical models in forecasting accuracy, while leaving the underlying physical explanations of atmospheric phenomena largely unresolved.
Without guarantees or interpretive frameworks, predictions risk becoming isolated facts rather than components of coherent understanding. In such cases, AI functions as an empirical instrument — powerful but epistemically incomplete. This limitation ensures that human judgment remains indispensable.
Looking Forward
The prospect of Artificial General Intelligence continues to animate debate. Capability has outpaced comprehension, and the gap shows no sign of closing automatically. What is clear is the enduring role of theorists and mathematicians. As computation becomes abundant, the bottleneck shifts toward conceptual clarity: identifying the right abstractions, formulating the right questions, and connecting empirical results to principled frameworks.
The critical task, therefore, is not to compute faster, but to think more carefully. AI makes finding answers easier; it does not tell us what is worth asking.
About the Author
Dinesh Karthik M, BS–MS 2027. Majoring in Mathematics with a minor in Data Science at IISER Pune, with an interest in theoretical machine learning.
Acknowledgement: The author sincerely thanks Devarshini M for her assistance in editing the final draft of the article.
References
- A. Vaswani et al., “Attention Is All You Need,” Advances in Neural Information Processing Systems, 2017.
- G. Hinton, “Geoffrey Hinton discusses promise and perils of AI at Toronto Tech Week,” University of Toronto News, 2024.
- R. S. Sutton, “The Bitter Lesson,” 2019.
- Nature Editorial, “Why AlphaFold earned the 2024 Nobel Prize in Chemistry,” Nature, 2024.
- T. Tao, “AI and Mathematics,” Talk at IMO 2024.
- Frontiers in Environmental Science, “AI in extreme weather events prediction and response,” 2025.