Two Compressions

Finding a pattern in observations is one achievement. Discovering the mechanism that generates it is another.

AI Can Find Kepler's Ellipses. It Cannot Yet Find Newton's Laws. This Essay Explains Why.

Core argument. There are two kinds of compression. The first takes raw data and finds the pattern: Brahe's planetary tables compressed into Kepler's ellipses. Current AI does this with extraordinary fidelity. The second takes the pattern and asks what generating mechanism would produce it as a necessary consequence: Kepler's ellipses explained by Newton's F = GMm/r². This essay argues that observational prediction within a fixed ontology underdetermines the second compression: proposing new state variables and generating mechanisms not already implicit in the observer's conceptual vocabulary. When AI systems appear to cross that boundary, they do so with added scaffolding: human priors, formal environments, interventions, search, verification, or all of the above. The argument proceeds through a distinction between the stream (observable output) and the kernel (an explanatory generative structure), then traces that distinction through Markov chains, Monte Carlo simulation, the Bellman equation, Bayesian inference, and the Causal Hierarchy Theorem. Each framework is powerful within a given ontological frame. None discovers a new one. Tool-augmented systems push the boundary by introducing feedback analogous to interventions, and the room beneath the ceiling is vast. But the second compression: proposing new state variables, new causal primitives, and new generating structures, remains a step that present systems reach only with substantial human scaffolding.

I. The Thought Experiment

Suppose you could build a large language model (LLM), one of the reasoning-capable systems now solving competition mathematics and writing production-grade code, and train it on everything humanity knew before 1687. It would have (i) Tycho Brahe's meticulous tables of planetary positions (Brahe, 1602); (ii) Johannes Kepler's three laws of planetary motion (Kepler, 1609), distilled from those tables through years of painstaking geometric analysis; and (iii) Galileo's kinematics, the medieval impetus theories, the entire corpus of ancient and Renaissance astronomy.

Then we give this LLM a prompt: find the simplest set of principles that explains all of this data.

Could it derive Newton's law of universal gravitation? Could it arrive at F = GMm/r²?

This is a more framed-up version of a question my colleague Prof Abdelrazek once asked me on a long walk. This is also a question that goes to the heart of what modern artificial intelligence can and (maybe) cannot do. The short answer is that the LLM is unlikely to arrive at Newton's laws. This essay presents the longer answer. The longer answer will argue that observational prediction within a fixed ontology underdetermines mechanism discovery, and that present AI systems which seem to cross that gap do so only with added scaffolding: human priors, formal environments, interventions, search, verification, or all of the above.

To see why, I want to make a distinction between two different kinds of compression, and then trace how that distinction propagates through key mathematical frameworks for learning from data: from Markov chains to Monte Carlo simulation, from the Bellman equation to Bayesian inference, and ultimately to recent research in causal hierarchy. The conclusion, this way, will be grounded rather than merely asserted.

II. The Two Compressions: Brahe to Kepler to Newton

Before presenting the formal argument, two terms need specific definitions. Both carry established technical meanings in computer science; I am using them in their most general form.

By kernel I mean the generating structure: the compact set of instructions from which an entire class of outputs can be produced. Think of it as the runnable abstraction, the function you could hand to a machine that would then produce the data indefinitely. In a Markov chain, the kernel is the transition matrix. In Newtonian mechanics, the kernel is the force law together with the equations of motion. In software engineering terms, the kernel is the source code, not the compiled binary's runtime behaviour.

By stream I mean the output sequence: the observable data, the traces, the predictions, the generated text, the list of states the system visits over time. The stream is what you can measure, record, and feed into a training pipeline.

A clarification on "kernel," because the term can mean several things. By "kernel" I do not mean the formally shortest bitstring on a universal Turing machine. I mean an explanatory generative structure: a compact set of state variables and transition laws that supports simulation, intervention, and counterfactual reasoning. The shortest program that reproduces a stream is not necessarily the best scientific explanation of it. Science cares about intervention, counterfactual support, unification, and explanatory transport, not just brevity. This distinction will matter when I formalise the kernel-stream gap in the sections that follow.

With these definitions in hand, the path from Brahe's data to Newton's laws involves two compressions, and they are different in kind. Making this distinction explicit is a core arc of the essay.

The first compression: Brahe to Kepler. Brahe's data is a table: positions of planets at times, thousands of careful observations. Kepler's laws are a compression: orbits are ellipses with the sun at one focus; the radius vector sweeps equal areas in equal times; the square of the orbital period is proportional to the cube of the semi-major axis. This is sophisticated curve-fitting with hypothesis selection. The output stream (planetary positions) is being described by a compact functional form. It is a compact description of the stream's statistical regularities, recoverable because the functional forms (ellipses, power laws) are within the mathematical vocabulary available to the observer.

A modern reasoning LLM with code execution, given Brahe's numerical tables and instructed to find the simplest mathematical description, could plausibly achieve this. Symbolic regression systems have already demonstrated comparable capabilities in analogous domains. The task is associational: finding patterns in observational data. It requires no causal reasoning, no counterfactual imagination, no extension of the hypothesis space. It requires only that the functional forms are within the model's representational capacity, which, for a system trained on the mathematical literature of the era, they would be. This is a first compression: stream to compact description of stream.

The second compression: Kepler to Newton. Newton's achievement (Newton, 1687) was not finding a better fit to the data. It was a unification: the same force that makes an apple fall makes the Moon orbit. The same law that explains Kepler's ellipses explains the tides, the precession of the equinoxes, the oblate shape of the Earth, and the perturbations between planets. This is a second compression: from compact description to generating mechanism. The first compression asks "what pattern does the data follow?" The second asks "what process would produce this pattern as its necessary consequence?" The answer to the first is in the stream. The answer to the second is not.

The second compression required several things that lie beyond what current AI systems do. (This is one plausible decomposition of Newton's achievement; historians might frame it differently, but the requirements I identify are individually well-motivated and collectively illustrate the boundary).

Extending the hypothesis space. The concept of a universal gravitational force did not exist in Kepler's framework. There was no prior probability to assign to it because there was no slot for it in the conceptual vocabulary. The hypothesis space had to grow to include a new mathematical object.

Identifying the right state space. Kepler's laws describe the output sequence: where is the planet? Newton's laws define the kernel: given position, velocity, and the masses, here is the acceleration. He discovered that the system is Markovian in phase space, in the joint space of position and momentum, with an inverse-square force kernel. That is not a pattern in the data. It is a statement about the causal structure of reality.

Counterfactual reasoning. Newton asked: what force law, acting on a body, would produce an elliptical orbit with the equal-area property? He worked backward from the observed pattern to the generating mechanism, then forward from the mechanism to predict phenomena that Kepler never observed. This is counterfactual reasoning at its most productive: reasoning about what the world would have to be like for the data to make sense. It is the inverse problem, recovering the kernel from the stream. (LLMs do engage in counterfactual reasoning in familiar domains. The claim is that Newton's counterfactual reasoning required proposing a new ontological category that did not exist in any prior description of the problem.)

Inventing the mathematical language. Newton had to create (or re-discover) the calculus, the language of differential equations, to even express the kernel. The representation did not exist. This is plasticity in its purest form: not finding a new value within an existing coordinate system, but creating a new coordinate system entirely.

The two-compression distinction frames what follows. First compressions (compact descriptions of statistical regularities) are recoverable from the stream by a sufficiently capable pattern-matcher. Kepler's laws are a first compression. Second compressions (generating mechanisms that explain why the regularities hold and predict new phenomena) are not recoverable from the stream alone, because they involve ontological categories not present in the observer's vocabulary. Newton's laws are a second compression. When I say "the kernel is not in the stream", I mean the second-compression kernel: the generating mechanism. First-compression kernels (compact descriptions) are in the stream, and AI systems find them with extraordinary skill.

The rest of this essay traces the two-compression distinction through the formal apparatus of probability, inference, and causal reasoning, then tests it against recent AI achievements.

III. Two Measures of Complexity

The kernel-stream distinction has a formal counterpart in information theory. A sequence has two distinct structural properties: (i) the unpredictability of its output, and (ii) the length of the shortest program that generates it. For the sequences that matter most to science, we are looking for a short generating law that can explain (potentially vast) seemingly irregular output.

Consider a pseudo-random number generator. A standard linear congruential generator takes the form x(n+1) = (a * x(n) + c) mod m, where a, c, and m are fixed constants. The output sequence looks indistinguishable from randomness. Statistical tests will find no pattern. By the measure of Shannon entropy (Shannon, 1948), the sequence has near-maximal complexity: each new number carries genuine surprise, and no amount of staring at the output helps you predict the next value better than chance. An engineer examining the stream alone would conclude it is incompressible.

But the object that generates the sequence is three numbers and an arithmetic operation. A single line of code. By the measure of Kolmogorov complexity (Kolmogorov, 1965), the system has tiny complexity. The generating program, the kernel, is trivially short.

Shannon entropy measures the statistics of the output stream. Kolmogorov complexity measures the length of the shortest program that generates the stream. The contrast is useful as a heuristic for the argument of this essay, but it is not a strict identity between shortest program and scientific explanation. The shortest program that reproduces a data stream might be an opaque lookup table or a brute-force encoding; science demands more than brevity. It demands explanatory structure: unification, counterfactual support, and transport to new domains. The Shannon/Kolmogorov distinction captures something real about the difference between first and second compression, but it captures it imperfectly. The second compression is not merely "find the shortest program". It is "find the explanatory generative structure".

The connection between Shannon's information theory (Shannon, 1948) and machine learning is not merely analogical; it is historical. The earliest neural networks designed to learn internal representations (from data stream), Hopfield networks (Hopfield, 1982) and Boltzmann machines (Ackley, Hinton, & Sejnowski, 1985), borrowed their mathematics directly from statistical mechanics. A Boltzmann machine has an energy function over its states, and learning adjusts the energy landscape so that the network's equilibrium distribution matches the data distribution. The restricted Boltzmann machine (RBM), which powered the deep learning revival of the mid-2000s (Hinton, Osindero, & Teh, 2006), made this tractable by restricting connectivity, but the training objective remained the same: minimise the divergence between the model's distribution and the observed data. In every case, the system learns to match the stream. The energy function is the internal machinery that achieves this matching; it is not, in general, the kernel that generated the data. Modern deep learning replaced energy-based training with gradient descent on cross-entropy loss, a more scalable descendant of the same Shannon-side objective. When an LLM trains on text, or on any sequential data, it is learning to assign high probability to sequences that appear in the training distribution and low probability to sequences that do not. It learns to compress the stream. It does not, in general, learn the kernel.

A valid objection arises here. Solomonoff's theory of inductive inference (Solomonoff, 1964) and Hutter's AIXI framework (Hutter, 2005) establish that a sufficiently optimal compressor will, in the limit, converge on the Kolmogorov-shortest program that generates the data. The convergence is real but requires enumerating all possible programs, a procedure that is uncomputable. No physical machine can perform Solomonoff induction. The gap between "theoretically bridgeable with infinite computation" and "practically bridgeable by gradient descent on cross-entropy" is the gap this essay is alluding to.

A second, more empirical objection: mechanistic interpretability research has shown that neural networks develop internal representations encoding structure beyond what the loss function targets (spanning syntactic trees, semantic relations, and rudimentary physical models). But there is a distinction between statistical representations that correlate with causal structure within the training distribution and genuine causal models that support intervention and counterfactual reasoning beyond it. Learned representations that mimic causal structure break under distributional shift precisely because they are sophisticated pattern matches, not generating mechanisms.

A natural conclusion from these objections would be that cross-entropy itself is the bottleneck. It is not. Cross-entropy is just likelihood-based training; with the right hypothesis class and the right data regime, it can recover hidden structure. The deeper issue is the conjunction of observational corpora, a fixed or human-supplied ontology, and the underdetermination of mechanism by observables. Prediction over observables does not, by itself, privilege the humanly meaningful mechanism. Shannon-optimal compression of statistical regularities is one of the most impressive engineering achievements in the history of computation. But it is not, by itself, sufficient for finding the generating program.

The Shannon/Kolmogorov distinction, however, remains abstract. To see why it matters concretely, for real physical systems, we need to trace it through the mathematical frameworks that scientists and engineers actually use to learn from sequential data. The most powerful of these are Markov chains (Markov, 1906), Monte Carlo methods (Metropolis & Ulam, 1949), the Bellman equation (Bellman, 1957), the Kalman filter (Kalman, 1960), and Bayesian inference (Bayes, 1763). If any of them can recover the kernel from the stream, the two-compression distinction would be a philosophical curiosity rather than a practical boundary. The next three sections test this.

IV. Markov's Insight: Structure from Minimal Assumptions

The kernel-stream distinction is sharpest in systems that evolve over time: a planet orbiting a star, a molecule diffusing in solution, a user navigating a website. At each moment, the process is in some state. The sequence of states it visits is the stream. The rule that governs transitions between states is the kernel. Markov chains are the natural starting point because they are the simplest and most general framework for such systems, and because, as we will see, Newton's laws turn out to be a Markov kernel: a compact transition rule in the right state space. Understanding why even this most natural framework cannot recover its own state space from output data alone is the key step in the argument.

Andrei Andreyevich Markov, in his 1906 study of dependent random variables, identified a structural condition under which such processes become tractable. The condition is often reduced to a slogan, "the future depends only on the present, not the past," but this understates what Markov actually conceptualised. His real insight was about extracting structure from data with minimal assumptions about dependence.

Before Markov, probability theory dealt almost exclusively with independent events. Markov showed that many of the same limit theorems hold for sequences with dependence, provided the dependence has a specific, disciplined structure: the current state captures everything relevant about the past for predicting the future. This is a statement about compression. The Markov property says: the entire history of the process can be compressed, without loss of predictive power, into a single object, the current state.

Back to the initial question about Newton's laws of motion. Newton's second law, F = ma, is a second-order differential equation. What it says, translated into the language of stochastic processes, is: given the position and velocity of a particle at time t, plus the forces acting on it, the entire future trajectory is determined. The system is Markovian, but only when you identify the right state variables. In the language of physics, the "state" is a point in phase space: the combination of position and velocity (or equivalently, position and momentum). Given the phase-space state at any instant, the laws of mechanics determine all future states. The Markov kernel, the function that maps the current state to the next, is the short program, but only once you have identified the right state variables. Finding the kernel and finding the state space are entangled: the representation itself is part of the hard problem, not a given. The kernel is the Kolmogorov-simple object hiding behind the Shannon-complex output stream, yet naming it presupposes that someone has already done the ontological work of choosing the right coordinates.

An observer watching the output stream of a Markov process can, with enough data, estimate the stationary distribution: how often the system visits each state. This is a Shannon-level quantity. With more sophisticated methods, the observer can estimate the transition probabilities themselves: given that the system is in state A, how likely is it to move to state B? This is better; it begins to approximate the kernel. But it requires that the observer already knows what the states are. If the state space is given, if someone tells you "the relevant variables are position and velocity," then estimating the transition probabilities from data is a tractable statistical problem. But identifying the state space itself, discovering that position and velocity are the right variables, that mass is the relevant property, that the inverse square of distance is the relevant functional form, is a categorically different problem.

A qualification is needed here, because statistical methods do discover latent state spaces. Hidden Markov models (Rabiner, 1989) infer hidden states from observations. Variational autoencoders (Kingma & Welling, 2014) learn compressed latent spaces. But they discover latent states within a pre-specified topology; the number of hidden states, the dimensionality of the latent space, or the structure of the model is given or searched over a pre-defined range. What Newton did was different in kind: he discovered that the relevant state space is phase space itself, a conceptual category (position-momentum pairs governed by forces) that did not exist in the pre-Newtonian vocabulary. Discovering latent variables within a given ontological frame is a statistical problem. Proposing that a new ontological frame is needed is not a statistical inference problem.

V. Within the Kernel: Monte Carlo, Kalman, and Bellman

If identifying the right state space is the hard problem, can powerful computational techniques project outward from data and generate new kernels? Three of the most consequential frameworks for operating within a kernel test this question: Monte Carlo simulation (Metropolis & Ulam, 1949) for exploring consequences, the Kalman filter (Kalman, 1960) for estimating hidden state, and the Bellman equation (Bellman, 1957) for deciding how to act. These represent three distinct modes of use: simulation, estimation, and control. If any computational framework could bootstrap its way from stream to kernel, these would be the strongest candidates.

Simulation. Monte Carlo methods (Metropolis & Ulam, 1949) explore the consequences of a known kernel by running it forward many times and averaging the results. They can estimate quantities that are analytically intractable and navigate high-dimensional spaces. But they require, as input, the kernel itself: the transition rule, the generating mechanism, the model from which the simulations are drawn. If you have Newton's laws, you can simulate a solar system. Monte Carlo does not discover the kernel. It exploits it.

Estimation. The Kalman filter (Kalman, 1960) tracks a hidden state by combining predictions from a state transition model with noisy observations. You hand the filter a set of matrices: the state transition model, the observation model, the noise covariances. These matrices are the kernel. If you define the state space wrong, the filter gives you the optimal estimate in the wrong variables. It will track. It will predict. It will not recover the dynamics it was never told to look for.

Control. The Bellman equation (Bellman, 1957) addresses what to do. In its most familiar form, it states that the value of being in a state s equals the maximum, over all possible actions, of the immediate reward plus the discounted expected value of the next state:

V(s) = max_a [R(s,a) + gamma * sum P(s'|s,a) V(s')]

This equation is the foundation of reinforcement learning, dynamic programming, and optimal control. But it takes as given: a state space S, a set of actions A, a transition kernel P(s'|s,a) specifying how the world responds to actions, and a reward function R(s,a) specifying what counts as success. The Bellman equation tells you the optimal policy given these ingredients. It does not tell you what the state space should be.

A natural objection: what about program synthesis and symbolic regression? Systems like DreamCoder (Ellis et al., 2021) search for compact programs from input-output examples, and symbolic regression tools like AI Feynman (Udrescu & Tegmark, 2020) discover equations from data. Both search for Kolmogorov-style objects rather than Shannon-style summaries. But program synthesis searches within a pre-specified domain-specific language, and symbolic regression searches within a given grammar of mathematical expressions. Both are powerful within their primitive vocabulary, and both take that vocabulary as given.

The pattern is consistent. Markov chains require a state space and a kernel. Monte Carlo requires a generative model. Kalman requires state transition matrices. Bellman requires states, transitions, and rewards. Program synthesis requires a language of primitives. Each framework is powerful within its given representation. None of them discovers the representation itself: the ontological categories that did not previously exist.

VI. Why Bayes Doesn't Save Us Either

At this point, a sophisticated reader might object: surely Bayesian inference (Gelman et al., 2013), the gold standard of rational belief updating, can bridge the gap? After all, Bayesian methods maintain probability distributions over hypotheses and update them in light of evidence. If we put Newton's law into the hypothesis space alongside other candidates, wouldn't the posterior eventually concentrate on the right answer?

The answer is: yes, if Newton's law is already in the hypothesis space. And that "if" is where the entire argument lives.

Bayesian inference is provably optimal for updating beliefs within a fixed hypothesis space. The Bernstein-von Mises theorem (van der Vaart, 1998) guarantees that, under regularity conditions, the posterior concentrates around the truth as data accumulates, provided the true model lies in the support of the prior. This is the key qualification. If the generating mechanism is not among the hypotheses you are considering, the posterior does not converge to the truth. It converges to the best approximation within your hypothesis space, the least wrong among the options you thought to include. This is the problem of model misspecification, and it is not a minor technical caveat. It is a fundamental issue.

Consider a pre-Newtonian astronomer attempting Bayesian inference on Brahe's data. The posterior would converge on whichever available hypothesis best fits the data, probably some well-tuned variant of Kepler's laws. It would never converge on F = GMm/r² because that hypothesis is not in the space. The concept of a universal gravitational force did not exist in the pre-Newtonian conceptual vocabulary. There was no parameter to tune. The hypothesis space itself had to be extended to include a new mathematical object.

Within standard parametric Bayesian inference, extending the hypothesis space is not a Bayesian operation. There is no prior over "all possible future concepts that have not yet been invented." Non-parametric Bayesian methods such as Dirichlet processes (Ferguson, 1973), and Bayesian program learning (Lake, Salakhutdinov, & Tenenbaum, 2015) do extend the hypothesis space as data arrives. But they extend along pre-specified dimensions. A Dirichlet process grows the number of clusters, but the notion of "cluster" and the distance metric are given. Bayesian program learning grows the complexity of programs, but the base primitives and the composition grammar are given. These methods explore more of a map that has already been drawn. What Newton did was different: he proposed an entirely new kind of object (universal gravitational force as a continuous vector quantity acting at a distance) that had no slot in any existing meta-space.

The act of proposing a fundamentally new kind of explanation, a new ontological category, a new mathematical structure, is what we might call plasticity or conjecture: the ability to redraw the map rather than merely navigate it, or to extend it along pre-specified dimensions. And it is this type of radical plasticity that current AI systems, whether Bayesian or frequentist, deep or shallow, parametric or nonparametric, lack.

Bayesian inference can estimate the transition kernel of a Markov chain if you give it the right state space. Nonparametric Bayes can discover the number of latent states. But if the true process operates in a state space built from ontological categories you have not identified, then Bayesian updating over your existing vocabulary will give you the best approximation in the wrong variables. It will fit. It will predict tolerably well. But it will not be the mechanism.

VII. The Causal Hierarchy as Constraint

The limitation described in the preceding sections might seem like a practical difficulty, something that cleverer engineering could eventually overcome. The Causal Hierarchy Theorem sharpens this intuition into a formal constraint: observational fitting alone is generally insufficient, and any successful system must get traction from something beyond bare association.

Judea Pearl's work on causality (Pearl, 2000; see also Pearl & Mackenzie, 2018) introduced the framework that covers this well. Pearl organised causal reasoning into three layers, each requiring fundamentally different kinds of knowledge. Bareinboim, Correa, Ibeling, and Icard (2022) then proved the theorem that gives this hierarchy its true value: the layers almost never collapse (described further below). The three layers are as follows.

The first layer is association: seeing. What correlates with what? Given that I observe X, how likely is Y? This is the layer of statistical patterns, of conditional probabilities, of "when Mercury is at this position, it tends to precess by this much." Every prediction a large language model makes is an associational inference. This is the domain of cross-entropy minimisation, pattern recognition, and statistical learning.

The second layer is intervention: doing. What happens if I change something? Not "when X occurs, what tends to follow?" but "if I were to set X to a specific value, holding everything else fixed, what would Y be?" This is Pearl's do-operator (Pearl, 2000): P(Y | do(X = x)) is fundamentally different from P(Y | X = x). The first requires a causal model, a representation of mechanisms, not merely correlations. The second can be estimated from observational data alone. Confusing them is the error behind every failed policy based on correlational evidence, and no amount of additional observational data resolves the ambiguity.

The third layer is counterfactual: imagining. What would have happened if things had been different? What kind of universe would make the Michelson-Morley null result (Michelson & Morley, 1887) unsurprising? What force law, acting on a body, would produce an elliptical orbit with the equal-area property? This is the layer at which Einstein (1916) operated when he asked not what the data showed but what the world would have to be like, at a fundamental level, for the data to be the natural and expected outcome. Counterfactual reasoning requires not just a causal model but a fully specified structural causal model (SCM), a set of deterministic functions and exogenous variables from which all lower-layer quantities can be derived. It is the most expressive, most demanding, and most scientifically productive level of reasoning.

The Causal Hierarchy Theorem (CHT), proved by Bareinboim et al. (2022), states that the three layers almost never collapse. For almost any structural causal model, the layers remain distinct. Data at one layer virtually always underdetermines information at higher layers. The "almost never" is precise: the set of SCMs for which the layers collapse has measure zero. The corollary: to answer questions at Layer i, one needs knowledge at Layer i or higher. No amount of associational data, no matter how much, no matter how carefully collected, yields interventional knowledge. No amount of interventional data yields counterfactual knowledge.

What does this imply for artificial intelligence? The CHT is a theorem about probability distributions over structural causal models, not about neural networks. Applying it to LLMs requires a bridging argument, and the bridge is imperfect. A system trained to minimise cross-entropy on observational text is learning the statistical structure of sequences: Layer 1 associations. The CHT says that Layer 1 data almost never determines Layer 2 or Layer 3 quantities. This does not prove that LLMs cannot engage in causal or counterfactual reasoning; what it does is make clear the sense in which observational data alone is generally insufficient. Any system that appears to cross the boundary must be getting traction from something beyond bare association.

An important counter-objection: LLM training data is not "observational data" in the simple sense the CHT assumes. The training corpus includes scientific papers describing experiments, textbooks explaining causal reasoning, and discussions of counterfactual scenarios. Can a system learn to reason causally by reading descriptions of causal reasoning? Reading about fire does not burn. A text describing an intervention is still observational data about an intervention. The system learns to predict what text about interventions looks like, which is a Layer 1 skill: pattern-matching over descriptions. The available evidence supports this: LLMs reproduce causal reasoning patterns from their training distribution but struggle with novel causal structures, precisely as the CHT would predict for a system that has learned the description of causal reasoning rather than acquiring causal access to the world. The same logic applies to emergent capabilities at scale: chain-of-thought reasoning, in-context learning, and tool use are more sophisticated patterns of Layer 1 inference, not transitions to Layer 2 or 3.

The deeper claim of this essay, however, goes beyond what the CHT strictly shows. The CHT establishes that observational data generally underdetermines higher-order causal questions. The essay's claim is about inventing new ontological categories and state variables: a meta-model problem adjacent to Pearl's hierarchy but not identical to it. The CHT is best understood here as a formal constraint that sharpens our understanding of why the gap exists, not as the complete proof that it cannot be closed.

VIII. Suggestive Episodes from the Frontier

Two episodes from early 2026 offer suggestive evidence of where the boundary between first and second compression falls in current practice. Neither is decisive on its own, but together they illustrate the pattern.

Knuth's Cycles. In February 2026, Donald Knuth published a note titled "Claude's Cycles" (Knuth, 2026), describing how Claude Opus 4.6, guided by his colleague Filip Stappers, solved an open combinatorics problem Knuth had been working on for weeks. The problem involved decomposing the arcs of a directed three-dimensional graph into Hamiltonian cycles, a hard combinatorial structure destined for a future volume of The Art of Computer Programming. Over 31 explorations spanning roughly an hour, Claude tried brute-force searches, invented geometric patterns, hit dead ends, changed strategies, and eventually converged on a construction, what it called a "serpentine pattern," that worked for all odd-numbered cases tested up to 101.

Knuth was impressed, calling the strategy "quite admirable," and noted: "It seems I'll have to revise my opinions about 'generative AI' one of these days." The division of labour is, however, more revealing (the interpretation I offer here is my own analytical frame, not Knuth's). Claude found the pattern: what this essay calls the first compression. Claude found a compact description of the combinatorial structure. Knuth wrote the proof: the mathematical induction showing why the construction must work for all odd values, the enumeration of all 760 valid solutions, the algebraic explanation of why the structure holds. The second compression, the generating mechanism, was done by Knuth. And when Stappers pushed Claude into the even case, where the problem's structure changes, the model degraded. Pattern-finding within a well-characterised combinatorial space is a first-compression task, and Claude performed it at a level that genuinely surprised a mathematician of Knuth's stature. Mechanism-finding, understanding why the pattern holds and adapting when the structure changes, is a second-compression task, and there the boundary was visible.

Knuth closed his note pointedly: "I think Claude Shannon's spirit is probably proud to know that his name is now being associated with such advances." It was Shannon he invoked, not Kolmogorov. The distinction was, I believe, deliberate.

The Anderson Conjecture. In April 2026, the AI4Math team at Peking University's Beijing International Center for Mathematical Research, led by Bin Dong in collaboration with algebraists Ruochuan Liu and Liang Xiao, announced that their automated AI framework had solved an open problem in commutative algebra, the Anderson Conjecture, proposed in 2014, and completed approximately 19,000 lines of formal verification in Lean 4 (Dong et al., 2026).

The architecture is more sophisticated than the Knuth case and pushes further against the ceiling. The team built two collaborating agents: Rethlas, handling natural-language mathematical reasoning, and Archon, handling formal verification. Crucially, they built a dual search-engine infrastructure, LeanSearch for the formalised mathematical library Mathlib, and Matlas for tens of millions of natural-language mathematical statements, that enabled cross-domain retrieval at a scale no human mathematician could match.

The breakthrough came when Rethlas, searching through Matlas, located a 2006 result by Jensen about completions of integral domains, a paper from an entirely different subfield, and recognised that it provided exactly the tool needed to construct a counterexample to Anderson's conjecture. Archon then formalised the proof, making two autonomous decisions along the way: rejecting its initial formalisation plan after detecting a logical gap, and finding an alternative route when a required concept was absent from Mathlib.

This work is genuinely impressive. The cross-domain retrieval, connecting Jensen's result to Anderson's problem, a connection no human had previously made, demonstrates AI search at superhuman scale. But the article's own authors are candid about the architecture. Liang Xiao "infused the agents' construction with a mathematician's modes of thinking, taste, and aesthetic judgment." The counterfactual and taste-level reasoning, specifically which problems to attack, what constitutes a productive direction, and when the framework itself needs questioning, came from a human mathematician. The formal verification (Lean 4) checks whether a proposed proof is internally valid. And the search, however vast, finds existing mathematics. It does not create new mathematical objects. The map was not redrawn. It was searched more thoroughly than any human could manage.

One productive interpretation: the composite system (human taste directing problem selection, AI search traversing the mathematical landscape, formal verification providing logical ground truth) solved a problem that neither humans nor AI could have solved alone. The system respects the causal hierarchy, with humans contributing Layer 3 reasoning, formal verification providing Layer 2 checking, and AI providing Layer 1 search. This division of labour suggests something beyond Kahneman's (2011) dual-process framework of fast intuition (System 1) and slow deliberation (System 2): a third mode in which human judgment sets the ontological frame and AI navigates within it at superhuman scale. One day it may be called System 3 mode.

IX. Convergent Formalisms

The kernel-stream distinction recurs independently across at least three major research programmes outside AI (yet, I belive to be adjacent and complementary), each approaching a structurally similar boundary from different starting points. The convergence is suggestive rather than conclusive, but the pattern is clear.

In chemistry and origin-of-life research, Cronin, Walker, and colleagues have developed assembly theory (Sharma et al., 2023), which assigns to any molecular object an assembly index: the minimum number of joining operations needed to construct it from basic building blocks. Molecules with high assembly index and high copy number are strong signatures of selection-driven processes, because only systems with memory (evolution, constructors) produce many copies of objects requiring long construction pathways. A mass spectrometer sees the molecule's properties, the stream. Assembly theory quantifies the construction pathway, the kernel.

In the philosophy of physics, Deutsch and Marletto's constructor theory (Deutsch & Marletto, 2015; Marletto, 2015) reframes physical laws not as dynamical equations but as counterfactual constraints on possible generating processes. The fundamental objects are not states and trajectories (the stream) but constructors: entities that can cause a transformation and retain the ability to do it again. Constructor theory asks what the space of possible kernels even is.

In theoretical neuroscience, Friston's free energy principle (Friston, 2010) describes living systems as entities that maintain a generative model by minimising variational free energy. The critical phrase is "given a generative model." The principle assumes the organism has a kernel and acts to keep its predictions aligned with incoming data. It is a theory of how systems maintain a kernel, not how they discover one. The origin of the kernel sits outside the framework, exactly as Newton's laws sit outside the Bellman equation that exploits them.

X. The Ceiling and the Room

The argument of this essay can now be stated as a single principle:

Current AI is exceptionally good at first compression: extracting compact predictive structure from observables. What it does not yet reliably do is second compression: proposing new state variables and generating mechanisms not already implicit in its ontology. Observational training alone underdetermines that step; when systems appear to cross it, they do so with added priors, interventions, or formal feedback.

A clarification is needed here, because deep learning's central achievement is representation learning. Convolutional networks (LeCun et al., 1998) discover edge detectors from raw pixels. Transformers (Vaswani et al., 2017) learn contextual embeddings from text. These are genuine discoveries. But they are statistical representations, compressions of the output stream optimised for prediction within the training distribution. They do not propose new state variables or new causal primitives. The distinction is between learning a better coordinate system for navigating existing data and proposing that the data has a generating structure nobody had previously imagined.

The ceiling, then, is real. The Causal Hierarchy Theorem provides the formal constraint for understanding it: data at one layer of the hierarchy virtually never suffices for inference at a higher layer. The problem is not "cross-entropy is insufficient". The problem is that prediction over observables, within a fixed ontology, does not by itself privilege the humanly meaningful mechanism. More data does not help. Larger models do not help. Longer context windows do not help. The limitation is structural, not quantitative.

But the most capable current systems are not bare language models. Coding agents that write code, execute it, observe compiler errors and test failures, and revise their approach are performing something structurally analogous to interventions: they change the system and observe consequences. I use "structurally analogous" deliberately, since Pearl's do-operator is formally defined over variables in a stochastic causal graph, and a compiler is a deterministic formal system, not a stochastic one. The analogy is not exact. But it is helpful: a type error is a proof of invalidity, not a correlation; a passing test suite is a logical guarantee within its scope, not a statistical association. The feedback that coding agents receive is closer to the logical structure of Layer 2 than to the statistical structure of Layer 1, and this helps explain why they are so much more effective than pure text predictors. The PKU team's Lean 4 verifier, the Knuth episode's executable search, every AI system that closes a loop through a formal environment: all of them derive their power from escaping pure Layer 1 via something that functions like a Layer 2 oracle.

The bound, however, remains: these agents operate within a given ontological frame. The programming language, the type system, the test suite, the proof assistant are the state space and the transition rules, provided by humans. A coding agent composes existing abstractions with extraordinary skill and formal feedback. It does not invent new programming paradigms, new type systems, or new formal languages. It navigates the map. It does not redraw it. (One might object that the line between "composing existing concepts in a novel way" and "inventing a new concept" is blurry. I address this in Section XI.)

The room beneath the ceiling is turning out to be vastly larger than anyone expected, and this is in a way the big bet motivating much of recent investment in AI. The Knuth episode shows that systems with formal feedback can navigate combinatorial landscapes with remarkable sophistication, finding patterns that eluded human experts. The PKU result shows that the right architectural scaffold (search engines covering millions of mathematical statements, formal verification against a ground-truth kernel, human taste guiding problem selection) can solve open research problems. These achievements are real and significant. They are the achievements of extraordinary navigation within a given ontological frame, not of rewriting the frame itself. But extraordinary navigation, properly scaffolded, can go very far indeed. In current organisations, and for a large category of problems, this may be sufficient.

What would it take to break through the ceiling? The answer likely requires at least genuine causal representations, not just associational ones. The PKU team's architecture hints at a second ingredient: formal verification as a ground-truth oracle, providing feedback that is not merely statistical but logically necessary. Assembly theory suggests a third: optimising for Kolmogorov-style construction pathways rather than Shannon-style output statistics. Perhaps the path forward involves all three: causal representations, formal verification loops, and Kolmogorov-oriented objectives. We do not yet know. But knowing why we are not there is the first step toward getting there.

XI. What This Means for AI Engineering

If the ceiling is real, what does it imply for practitioners? The answer requires reflecting on two kinds of abstraction that are easily conflated.

The first kind is compositional abstraction: organising known concepts into clean interfaces, well-factored modules, coherent APIs, and maintainable architectures. This is what a skilled software engineer does when designing a system, and it is what AI coding assistants now do. When an AI coding tool generates a well-separated service layer, identifies the right design patterns for a domain, or proposes a clean type hierarchy, it is performing compositional abstraction within a known vocabulary of software constructs. The building blocks (functions, types, protocols, modules) already exist. The task is to combine them well. This is sophisticated navigation of a well-mapped space, and current systems do it at a level that augments engineering productivity.

The second kind is ontological abstraction: identifying the right entities, the right state variables, the right decomposition of reality into categories that did not previously exist. When Newton proposed "force" as a continuous vector quantity acting instantaneously at a distance, he was not combining known concepts. He was introducing a new ontological category. When Darwin proposed "natural selection," he was not assembling existing biological concepts into a clean hierarchy. He was creating a new causal primitive. When Shannon defined "information" as a measurable, substrate-independent quantity, he was not refactoring existing communication theory. He was founding a new field by identifying a new kind of object.

The distinction maps onto the kernel-stream boundary and onto the two-compression framework. Compositional abstraction rearranges the stream: it finds better ways to organise, express, and combine what is already known. This is first-compression work. Ontological abstraction proposes a new kernel: a new generating principle, a new set of state variables, a new Markov structure in a space that nobody had previously identified. This is second-compression work.

I should acknowledge that the boundary between these two is fuzzy. At the extremes, the distinction is clear: designing a microservice architecture is compositional; proposing "force" as a new ontological category is ontological. But there is a middle territory where novel combinations of existing concepts produce something that feels genuinely new. The MapReduce paper (Dean & Ghemawat, 2004) is one such example. The operations "map" and "reduce" both existed as programming primitives. What Dean and Ghemawat recognised was that these two operations, composed in a specific way, characterise a vast class of distributed computations, a generating structure that nobody had previously identified and popularised. The primitives were known; the recognition that they form a generative pair for distributed computing was new. This is closer to ontological abstraction than to compositional abstraction, but it involved combining existing concepts rather than inventing from whole cloth. The extreme cases are clear enough to guide practical decisions, even if the boundary is fuzzy.

One might object that program synthesis research (Section V) is explicitly designed to search for compact generating programs: kernels, in this essay's terminology. But as argued there, program synthesis searches within a pre-specified language of primitives. It can find novel compositions of known primitives, which places it squarely in the MapReduce-style middle territory. DreamCoder's library learning pushes furthest: it grows its vocabulary by abstracting over successful compositions. But abstracting over compositions of existing primitives is not the same as proposing a primitive that no existing grammar could generate.

The practical implication is that the kind of abstraction AI performs is different from the kind that drives paradigm shifts, and the difference, at the extremes, is not one of degree. An AI system can generate an excellent repository structure for a microservices architecture because the space of design patterns has been extensively documented across millions of public codebases. It is less likely to generate the equivalent of the MapReduce paper, which required recognising a new generating structure. The first is assembling known parts skillfully. The second is recognising that a new part is needed, one that does not yet exist in anyone's vocabulary.

For AI engineers, this suggests a specific division of labour. Delegate compositional abstraction aggressively: system design, code generation, refactoring, documentation. These are navigation tasks, and the navigator is now superhuman in speed and breadth. Retain ontological abstraction as the core human contribution: deciding what entities the system should model, what state space captures the domain, what decomposition of the problem will still make sense when the requirements change in ways nobody anticipated. The former is the stream. The latter is the kernel. And writing the kernel, for now, remains a human act.

XII. Return to the Observatory

Brahe stood on the island of Hven, night after night, recording the positions of the planets with instruments of his own design, accumulating the most precise astronomical dataset the world had ever seen. Kepler inherited those tables and, through years of calculation, found the ellipses hiding in the numbers: a first compression of the output stream so elegant that we are still talking about it.

Newton did something different. He asked: what transition rule, what Markov kernel in phase space, would generate those ellipses as its inevitable consequence? And from that question, asked not of the data but of the universe, everything followed. The tides, the comets, the shape of the Earth, the precession of the equinoxes, and eventually, centuries later, the bending of starlight around the sun and the waves in spacetime detected by LIGO.

The LLM, given Brahe's data, would find the ellipses. It would find the area law, the period relation. It would master the first compression with a fidelity Kepler could only have dreamed of.

It would not find universal gravitation.

Because the second compression, the generating mechanism, is not in the stream. The transition rule is not in the output sequence. The process that produces the pattern is not deducible from the pattern alone. And the deepest structure of the world, as best we understand it, lives in those generating mechanisms: in the kernels that produce the streams we observe.

References

Ackley, D.H., Hinton, G.E., & Sejnowski, T.J. (1985). "A Learning Algorithm for Boltzmann Machines." Cognitive Science, 9(1), 147-169.

Bareinboim, E., Correa, J.D., Ibeling, D., & Icard, T. (2022). "On Pearl's Hierarchy and the Foundations of Causal Inference." In Geffner, H., Dechter, R., & Halpern, J. (Eds.), Probabilistic and Causal Inference: The Works of Judea Pearl. ACM Books, pp. 507-556.

Bayes, T. (1763). "An Essay towards Solving a Problem in the Doctrine of Chances." Philosophical Transactions of the Royal Society of London, 53, 370-418.

Bellman, R. (1957). Dynamic Programming. Princeton University Press.

Brahe, T. (1602). Astronomiae Instauratae Progymnasmata. Prague. Published posthumously by Kepler.

Dean, J. & Ghemawat, S. (2004). "MapReduce: Simplified Data Processing on Large Clusters." OSDI '04: Proceedings of the 6th Symposium on Operating Systems Design and Implementation, pp. 137-150.

Deutsch, D. & Marletto, C. (2015). "Constructor Theory of Information." Proceedings of the Royal Society A, 471(2174): 20140540.

Dong, B. et al. (2026). "AI4Math: Automated Solution of the Anderson Conjecture." Technical report, Beijing International Center for Mathematical Research, Peking University. Available at bicmr-ai4math.github.io.

Einstein, A. (1916). "Die Grundlage der allgemeinen Relativitätstheorie." Annalen der Physik, 354(7), 769-822.

Ellis, K., Wong, C., Nye, M., Sable-Meyer, M., Morales, L., Hewitt, L., Cary, L., Solar-Lezama, A., & Tenenbaum, J.B. (2021). "DreamCoder: Bootstrapping Inductive Program Synthesis with Wake-Sleep Library Learning." Proceedings of the 42nd ACM SIGPLAN International Conference on Programming Language Design and Implementation (PLDI '21), pp. 835-850.

Ferguson, T.S. (1973). "A Bayesian Analysis of Some Nonparametric Problems." The Annals of Statistics, 1(2), 209-230.

Friston, K. (2010). "The Free-Energy Principle: A Unified Brain Theory?" Nature Reviews Neuroscience, 11(2), 127-138.

Gelman, A., Carlin, J.B., Stern, H.S., Dunson, D.B., Vehtari, A., & Rubin, D.B. (2013). Bayesian Data Analysis (3rd ed.). Chapman and Hall/CRC.

Hinton, G.E., Osindero, S., & Teh, Y.W. (2006). "A Fast Learning Algorithm for Deep Belief Nets." Neural Computation, 18(7), 1527-1554.

Hopfield, J.J. (1982). "Neural Networks and Physical Systems with Emergent Collective Computational Abilities." Proceedings of the National Academy of Sciences, 79(8), 2554-2558.

Hutter, M. (2005). Universal Artificial Intelligence: Sequential Decisions Based on Algorithmic Probability. Springer.

Kahneman, D. (2011). Thinking, Fast and Slow. Farrar, Straus and Giroux.

Kalman, R.E. (1960). "A New Approach to Linear Filtering and Prediction Problems." Journal of Basic Engineering, 82(1), 35-45.

Kepler, J. (1609). Astronomia Nova. Prague.

Kingma, D.P. & Welling, M. (2014). "Auto-Encoding Variational Bayes." Proceedings of the 2nd International Conference on Learning Representations (ICLR 2014).

Knuth, D.E. (2026). "Claude's Cycles." Stanford Computer Science Department, 3 March 2026, revised 4 March 2026. Available at cs.stanford.edu/~knuth/papers/claude-cycles.pdf.

Kolmogorov, A.N. (1965). "Three Approaches to the Quantitative Definition of Information." Problems of Information Transmission, 1(1), 3-11.

Lake, B.M., Salakhutdinov, R., & Tenenbaum, J.B. (2015). "Human-Level Concept Learning through Probabilistic Program Induction." Science, 350(6266), 1332-1338.

LeCun, Y., Bottou, L., Bengio, Y., & Haffner, P. (1998). "Gradient-Based Learning Applied to Document Recognition." Proceedings of the IEEE, 86(11), 2278-2324.

Markov, A.A. (1906). "Extension of the Law of Large Numbers to Dependent Quantities." Izvestiya Fiziko-Matematicheskogo Obshchestva pri Kazanskom Universitete, 2nd series, 15, 135-156.

Marletto, C. (2015). "Constructor Theory of Life." Journal of the Royal Society Interface, 12: 20141226.

Metropolis, N. & Ulam, S. (1949). "The Monte Carlo Method." Journal of the American Statistical Association, 44(247), 335-341.

Michelson, A.A. & Morley, E.W. (1887). "On the Relative Motion of the Earth and the Luminiferous Ether." American Journal of Science, 34(203), 333-345.

Newton, I. (1687). Philosophiae Naturalis Principia Mathematica. London: Royal Society.

Pearl, J. (2000). Causality: Models, Reasoning, and Inference. Cambridge University Press.

Pearl, J. & Mackenzie, D. (2018). The Book of Why: The New Science of Cause and Effect. Basic Books.

Rabiner, L.R. (1989). "A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition." Proceedings of the IEEE, 77(2), 257-286.

Shannon, C.E. (1948). "A Mathematical Theory of Communication." Bell System Technical Journal, 27, 379-423, 623-656.

Sharma, A., Czegel, D., Lachmann, M., Kempes, C.P., Walker, S.I., & Cronin, L. (2023). "Assembly Theory Explains and Quantifies Selection and Evolution." Nature, 622, 321-328.

Solomonoff, R.J. (1964). "A Formal Theory of Inductive Inference." Information and Control, 7(1), 1-22; 7(2), 224-254.

Udrescu, S.M. & Tegmark, M. (2020). "AI Feynman: A Physics-Inspired Method for Symbolic Regression." Science Advances, 6(16), eaay2631.

Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, L., & Polosukhin, I. (2017). "Attention Is All You Need." Advances in Neural Information Processing Systems 30 (NeurIPS 2017), pp. 5998-6008.

van der Vaart, A.W. (1998). Asymptotic Statistics. Cambridge University Press.