Rethinking Scaling

In “A Proposal For the Dartmouth Summer Research Project on Artificial Intelligence”, the authors based the proposed research program on the central conjecture that “every aspect of learning or any other feature of intelligence can in principle be so precisely described that a machine can be made to simulate it.” Today’s machines have successfully exhibited several aspects of intelligence – from language production to visual perception – and these successes invite us to reconsider what a “precise description” entails.

Recent advances in large language models (LLMs), vision-language models (VLMs), and diffusion models suggest that it is possible to engineer systems with strikingly human-like capabilities without teaching them to learn in human-like ways. The success of a statistical approach to learning has reignited the long-standing debate on the value of explanation versus prediction for advancing science, and more importantly, inspired sweeping claims about what these systems reveal about intelligence.

One of the most dominant claims to emerge from these successes concerns the role of scale. The easy narrative about AI goes: data is all you need. As increasingly large models trained on increasingly large datasets continued to improve, some commentators announced the “end of theory” and treated scale as the primary driver of future progress. Chris Anderson’s claim that “we can stop looking for models” and instead “throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot” captures the most extreme version of this sentiment.

Modern AI constitutes the largest experiment ever performed on intelligence. But when we reflect on the progress made, the narrative that data played the most paramount role is all too reductive. Scaling has been the right strategy when data, compute, and model size offered reliable returns. But its success should not be mistaken for evidence that scale is the only source of progress. As these resources become more constrained, the central question changes: given a fixed budget, what properties must a learning system have in order to learn efficiently and generalize? The history of NLP—and the current challenges in post-training—suggests that answering this question requires more than scaling.

Engineering successes and failures can help us answer this question. Successful systems can weaken claims of necessity—for example, the claim that linguistic competence requires an explicitly specified symbolic grammar. Yet they do not tell us which other approaches might also work, whether one approach is best, or which is most resource-efficient. On the other hand, failed endeavors falsify claims of sufficiency, demonstrating that what is already in the system might not be sufficient to produce the observed capability. But failure does not rule out an approach altogether without carefully controlled experiments and ablations: the relevant limitation may lie in the data, representation, objective, architecture, or scale at which it was tested.

Is Data All You Need?

The availability of data for training has led to major breakthroughs in natural language processing and computer vision, leading many researchers to endorse the data scaling hypothesis – the idea that “the quality and quantity of data used to train an AI model is by far the most important determinant of model performance.” “Data is All You Need” and “Data is The New Oil” are two equivalent ways of expressing the same idea.

Claims about the importance of data are not unwarranted. In fact, much empirical evidence shows that scaling law is a key property of transformers: cross-entropy loss in language models scales as a power-law with dataset size (along with model size and compute). The success of increasing training data has encouraged bitter-lesson-like claims about data and compute, that “the only thing that matters in the long run is the leveraging of computation.” These claims endorse the value in what has shown to scale – data and compute – and cast doubts on methods whose returns do not appear to grow as reliably: for example, that “the two methods that seem to scale arbitrarily …are search and learning.”

When we glance at the history of language modeling, it might seem like scale is all we need, but the full picture is a lot more complicated: progress has depended not only on increasing data and compute, but on identifying when a learning system lacked a property necessary to use those resources effectively. The current post-training landscape echoes a similar trend, when training data becomes secondary to algorithmic improvements.

History of NLP: a co-evolution of theory and engineering

First, let’s consider the history of language modeling. The succession of approaches that paved the way for today’s LLMs is also a history of refining hypotheses about what a computational solution to language must satisfy: generalize beyond training data, use context, and capture long-range dependencies. It was not simply a matter of increasing data and compute. Progress emerged through a co-evolution of theory and engineering, in which computational hypotheses guided new designs, and the failure modes of those designs revealed which properties a current solution still lacked.

N-gram models offered an early and elegant way to bring statistical regularities into language modeling. By estimating the probability of a word from the preceding words, they could capture local context, but they quickly encountered a sparsity problem: even large corpora contain many unseen word sequences, making reliable estimates difficult as the context window grows. Early neural language models addressed part of this rigidity by learning distributed word representations; however, these models typically retained a fixed-size context window and could not directly condition on information farther back in a sequence. These early systems revealed both the promise and the limits of statistical learning. More data could alleviate sparsity, but it couldn’t resolve the architectural limit on how much preceding context the model could access. The next advance therefore required not simply more resources, but a different mechanism for representing and using longer-range context.

Then came RNN, which could process any input length and could in theory use information from previous steps, but unfortunately the vanishing gradient problem made it difficult to access information from many steps back. When gradients are too small, the signal shrinks as it backpropagates; in practice, we see that models cannot answer reference resolution questions about texts that appeared a couple sentences ago. This failure revealed a more specific requirement: an effective language model needs a mechanism for selectively preserving and retrieving relevant information over long contexts. Long short-term memory networks (LSTMs) addressed part of this problem through a memory cell and gates that regulate what information is written, retained, exposed, and forgotten. This mechanism allowed models to learn dependencies across longer spans.

Self-attention allows each position to directly weight and aggregate information from other positions in the sequence, and positional embeddings in the transformer supply information about order. This gave the model a more direct, content-dependent way to access distant tokens and also allowed much greater parallelization during training. Its success in the language modeling task isn’t just a story about scaling but more about how to translate refined computational theories into engineering solutions.

It shows us, concretely, that language modeling requires the following components: 1. learning statistical regularities from linguistic input, 2. Representing the order of language, 3. Integrating information across context, and 4. Learning how different parts in the context relate to each other. Each generation of language models—from n-grams, recurrent networks, and Transformers—therefore represents not merely a better algorithm, but a refinement of our understanding of the computational problem itself.

Data-efficient learning

Even though LLMs can perform many tasks as humans and can even outperform the average human on some tasks, they learn from very different mechanisms. One of the biggest differences is the amount of data that we learn from. Modern LLMs are trained on orders of magnitude more text than any individual human encounters in a lifetime, yet we are able to learn language and gain other cognitive capabilities – through a much more data-efficient way! This insight itself is valuable: we are the living evidence that it is possible to learn with so much less.

Research on data curation and data mixtures shows that what models learn depends not only on the amount of pretraining data, but also on its composition and quality. The BabyLM challenge studies language learning under human-scale data budgets, demonstrating that architectural choices, training objectives, and interaction can produce substantial gains in constrained regimes – and that data-limited settings can reveal meaningful differences between competing approaches to learning. Work on pre-training under infinite compute makes a similar point: it shows that with limited data, there are many other things to be leveraged – epoching, regularization, parameter scaling, model ensembling, etc. The paper operates under an interesting thought experiment: if data is limited, what else is there to leverage? This question wasn’t a priority in the ML community for a long time. When larger datasets reliably delivered clear gains, increasing data was often the quickest and most tangible route to better performance. Why not do it?

LLM post-training in the age of limited data

Now, this question has become extremely relevant (at least in the NLP community) because there simply aren’t much text data left for models to learn from. There’s been a growing sentiment of an end of the scaling era. The question worth asking is changing from how much more data do we need for a reasoning model to do X to: assuming a fixed dataset budget, how can we get a reasoning model to do X? And indeed, from the recent progress in LLM post-training has proven that performance can depend substantially on algorithmic choices even when the base model, task distribution, and verifier are held broadly fixed. GRPO, DAPO and other variants differ in how they estimate advantages, clip policy updates, sample responses, assign loss across tokens, and manage training stability. These differences show that, under a constrained feedback setting, the learning procedure can determine how effectively a model uses the learning signals it receives. Other recent work pushes this idea further: proRL shows that dynamically controlling entropy in training is key; RLP treats a sampled chain of thought as an exploratory action and rewards it according to whether it improves prediction of the next token. This creates a dense, verifier-free signal from ordinary text, rather than requiring an additional corpus of externally labeled reasoning traces. On-policy self distillation further supports the possibility of learning more efficiently without needing more external data: we can simply use information-gain (or privileged information) itself as a reward!

One way to see why learning mechanisms matter is the difference between supervised fine-tuning (SFT) and reinforcement learning. SFT can be understood as minimizing a forward-KL objective toward a target distribution, whereas RL induces a more mode-seeking, reverse-KL-like update. These differences can lead to different empirical trade-offs: the finding that SFT memorizes and RL generalizes shows directly that some algorithms can be fundamentally limited in what they can do and that two procedures trained on similar samples can produce meaningfully different learning dynamics and generalization behavior.

This important distinction is not just a matter of learning efficiency. It pertains to fundamental questions about machine understanding. What has a model learned when it succeeds? A model that has consumed an unimaginable amount of data might succeed at particular benchmarks because it learned certain shortcuts (e.g. pattern matching) but might fail catastrophically when the benchmarks measure true competence and not just mere performance (there’s plenty of evidence for this in the interpretability literature!) As Melanie Mitchell puts it, models that memorize only learned “statistical correlations that enable near-perfect performance without humanlike understanding.” A brute-force scaling approach (without considering whether the algorithms are aligned with the computational objective) can easily lead to models that succeed at benchmarks but fail on tasks where memorization does not suffice, where models need to generalize in out-of-distribution domains.

When progress comes from indiscriminately adding data and compute, it can be difficult to tell why a system improved. And there was just the easy narrative: yay to scaling. On the other hand, when progress comes from controlled changes to the learning process under a fixed resource budget, those differences become evidence: they let us ask what information, objective, or inductive bias produced a capability. This does not necessarily make the resulting networks transparent, but it makes the path to capability more scientifically interpretable.

The current language-modeling landscape offers a valuable lesson for other domains of intelligence. Scaling can deliver large and reliable short-term gains, but it does not tell us which learning mechanisms will remain effective once a relevant resource becomes scarce. In robotics, for example, the constraint might be diverse, grounded interaction data: safe physical experience is expensive to collect, difficult to represent, and cannot simply be scraped from the web. The question is therefore not whether to abandon scaling, but how to identify the bottleneck that scale alone cannot remove—and to use that constraint to develop and test better ways of learning.

Other Considerations

Data scarcity is only one way scaling can cease to be the answer. In many domains, the deeper challenge is representational: the relevant structure of the world may not yet be available in a form that learning systems can use. Language and images have benefited from relatively standardized digital representations—text tokens and pixels—but other forms of experience, such as touch, smell, or embodied interaction, are not so easily captured in datasets that preserve this structure. And some targets—dynamic human behavior, open-ended environments, or stochastic processes—may remain uncertain even with ideal data and models. The next era of AI progress may therefore depend less on scaling a single recipe than on identifying which constraint a problem actually presents. The history of NLP offers a model for this: we can treat failures not as evidence that learning has reached its limit, but as clues about the data, representations, and computational properties a successful system still lacks.