Intelligence Beyond the Next Token

The useful structure inside a model can be richer than the output channel through which we ask it to speak.

AI systems already know more than they show; the opportunity is in connecting that hidden knowledge to the world

A technical companion to the essay series "The Agent, Not the Task" and "The Mapmaker's Alphabet"

Core thesis. When AI systems learn to predict the next word, they build internal models of the world far richer than the text they produce. The intelligence lives in those hidden structures, not in the outputs we see. What determines their quality is not scale alone but the mathematical coherence of the architecture that constructs them. And these systems already know more than they can show & the immediate opportunity is in connecting that hidden knowledge to the world.

Abstract. When AI systems learn to predict the next word, they quietly build something much more valuable: compressed internal models of the world that go far beyond fluent text. This note argues that three principles determine what these systems can achieve, regardless of their size. First, it is the prediction task itself that forces the system to build useful internal structure; and the quality of that structure, not the model's scale alone, determines what it can do. Second, architectures built on the right mathematics consistently outperform larger ones assembled from engineering intuition, because the mathematics makes explicit what the intuition was already reaching for. The boundary between mathematical insight and engineering craft is blurry and productive in both directions, and not every mathematical idea succeeds. Third, these systems already know more than they show. Expanding what they can act on (through tools, code execution, and real-world feedback) unlocks capabilities that were already present but had no way out. The long-term investment is in better mathematical foundations; the immediate opportunity is in building better bridges between what these systems know and what they can do.

A note on terminology: This companion note uses "machine intelligence" to refer to the computational capabilities exhibited by trained systems (language models, transformers, and their derivatives). Where the note discusses "intelligence" without the qualifier, it is drawing on the broader theoretical lineage (Shannon, Kolmogorov, Schmidhuber, Ma et al.) that applies to both biological and computational systems. The companion essays develop this dual usage in detail; the intelligence quartet (Chollet, Levin, Ma, Bennett) is deliberately architecture-agnostic, while the three-part constraint identifies where current machine architectures fall short of biological ones.

1. The Accidental Discovery

In 2017, Alec Radford, Rafal Jozefowicz, and Ilya Sutskever at OpenAI trained a character-level language model on 82 million Amazon product reviews (Radford, Jozefowicz, and Sutskever, 2017). The task was simple: predict the next character. No labels, no supervision, no objective beyond sequence prediction. When the researchers examined the model's internal representations, they found something unexpected. A single neuron in the 4,096-unit LSTM had learned to track sentiment, achieving 91.8% accuracy on the Stanford Sentiment Treebank despite never having been trained on sentiment at all. Manipulating that neuron's activation allowed the model to generate text with controlled emotional valence. The model had discovered that understanding sentiment was useful for predicting what comes next, and so it built a detector as a byproduct of compression.

This was more than a curiosity. It was the first clear evidence that next-token prediction, treated as a pure compression objective, drives the formation of structured internal representations that capture high-level semantic properties of the domain. The model did not need to be told what sentiment was. It needed to predict characters, and sentiment turned out to be a regularity worth compressing. The finding directly shaped what came next: Radford went on to lead the development of GPT-1 (Radford et al., 2018) and GPT-2 (Radford et al., 2019), scaling the same insight from a single neuron to billions of parameters. The philosophical foundation was already in place: if next-token prediction on diverse text produces useful semantic representations at small scale, what happens at large scale?

What happened was that the representations became richer, more structured, and more surprising. But for several years, the field's understanding of what the models had learned remained under-explored, focused on behavioural benchmarks (what the model could do) rather than mechanistic explanations (how it did it). The journey from the sentiment neuron to the present has been a progressive deepening of that mechanistic understanding, and the story it tells challenges the prevailing treatment of the token sequence as the primary window into intelligence in these systems.

2. From Features to World Models

The sentiment neuron showed that a language model can learn a single interpretable feature as a compression byproduct. The question that followed was more ambitious: can models learn not just isolated features but structured models of the processes that generate their training data?

Li et al. (2023) provided the clearest evidence. They trained a GPT-variant on Othello game transcripts (sequences of legal moves) and found that the model had learned an internal representation of the board state itself. The model was never shown a board. It processed only move sequences. Yet nonlinear probes applied to its hidden states could recover the full board configuration with high accuracy, and intervening on these internal representations causally changed the model's predictions about legal moves. A follow-up study (Li et al., 2023b) revealed that these board-state representations are encoded linearly in the model's activations, meaning the world model is not buried in complex nonlinear circuits but is geometrically accessible, recoverable through simple linear projections.

This finding generalised a principle implicit in the sentiment neuron work: next-token prediction does not merely learn surface statistics. It builds compressed models of the data-generating process because such models are useful for prediction (an observation that Ilya Sutskever reflects on consistently across multiple talks). A model that tracks sentiment predicts review text better. A model that tracks board state predicts legal moves better. The compression objective drives the emergence of structured internal representations that capture the causal and relational structure of the domain, not because the model was told to learn that structure, but because learning it reduces prediction error.

The implication is suggestive, but the analogy requires care. Othello is a closed, deterministic, fully observable domain; natural language is open-ended, ambiguous, and generated by processes far more complex than board-game rules. The principle that transfers is not the specifics of what was learned but the mechanism by which it was learned: compression of sequential data drives the formation of structured internal models of the data-generating process. The sentiment neuron (§1) already demonstrated this mechanism in the language domain at small scale — a single semantic feature emerging from character prediction — and Li et al.'s linearity result shows that the resulting representations are geometrically structured, not buried in opaque nonlinear circuits. Mechanistic interpretability research on language models confirms that the principle scales: large language models build compressed representations of the processes that produce human text, including semantic relationships, causal patterns, argumentative structures, domain-specific regularities, and cross-domain isomorphisms. The representations are richer and more heterogeneous than Othello's board state, precisely because the data-generating process is richer. These are the representations that "The Mapmaker's Alphabet" calls the binding substrate: the geometric space in which statistical similarity encodes (as a proxy) structural similarity across domains.

3. Inside the Machine: Circuits, Induction, and Latent Computation

Mechanistic interpretability has progressively revealed the computational structures through which these representations operate.

Elhage et al. (2021) introduced a mathematical framework for understanding transformer circuits, showing that the computation in attention layers can be decomposed into interpretable components. This was foundational: it established that transformers are not opaque black boxes but systems whose internal operations can, in principle, be reverse-engineered into understandable computational motifs.

Olsson et al. (2022) identified induction heads, a specific circuit motif in which pairs of attention heads implement a form of in-context pattern matching. One head identifies a pattern in the input; the other copies the continuation of that pattern to the output. This mechanism underlies much of what appears as in-context learning (ICL): the model does not merely condition on previous tokens as static context but dynamically activates computational pathways based on patterns detected in the input. The researchers showed that these circuits form during training in a sudden transition (a "phase change") rather than emerging gradually, and that they account for much of the in-context learning capability observed in large models.

Von Oswald et al. (2023) deepened this by demonstrating that in-context learning (ICL) can be understood as implicit gradient descent within the model's latent space. When the model processes a prompt containing examples, it effectively performs a form of meta-learning: inferring the task from the examples and adapting its internal computation accordingly, all within a single forward pass. The emitted tokens are downstream artifacts of this latent adaptation, not the process itself.

Together, these findings reveal a picture of transformer computation that is fundamentally different from the naive "pattern completion over tokens" view. The model implements a collection of distributed computational mechanisms (circuits) that are dynamically composed during inference. Multiple mechanisms may be active simultaneously, competing or cooperating within the model's high-dimensional latent space. The final output is the result of a selection process over interacting internal trajectories, not a single trace being followed.

4. Beyond Language: Latent Reasoning and Hidden Features

The evidence so far concerns what models represent. A natural objection is that representation is not reasoning. Kambhampati et al. (2024) press this point hardest: their LLM-Modulo framework argues that autoregressive models cannot truly plan or self-verify, that what appears as reasoning is pattern completion over plausible token trajectories, and genuine reasoning requires external search, symbolic validation, and feedback loops. Follow-up work from the same group reinforces the critique. Kambhampati et al. (2025) warn explicitly against treating intermediate tokens as "thinking," arguing that chain-of-thought traces are not faithful explanations of the model's computational process. Bhambri, Biswas, and Kambhampati (2025) show that models trained to produce interpretable reasoning traces can behave unpredictably, suggesting that traces are artifacts of training signals rather than windows into the model's actual reasoning.

This critique is right about an important point: the token trace is not where reasoning lives. But the conclusion that reasoning is therefore absent does not follow from the evidence presented. What the critique demonstrates is that token traces are unreliable windows: they do not faithfully represent whatever computational process produced them. That finding is consistent with two possibilities — either the model performs no genuine reasoning at all, or it reasons through mechanisms that are not visible in the trace. The latent reasoning evidence reviewed below provides direct evidence for the second possibility: structured computation that operates in the model's internal representational space, independent of (and sometimes invisible to) the token sequences it emits.

Hao et al. (2025) introduced Coconut (Chain of Continuous Thought), showing that models can maintain and evolve internal reasoning states directly within their hidden representations, without projecting intermediate steps into tokens. These latent trajectories can encode multiple candidate paths in parallel, enabling forms of implicit search that are impossible under the constraints of sequential token generation. Models trained with continuous thought achieved comparable or better results than chain-of-thought methods while using significantly fewer tokens, demonstrating that the reasoning process is more efficient when freed from the bottleneck of natural language expression.

Most strikingly, Rofin, Naghiyev, and Hahn (2026) demonstrated that transformers learn features that are entirely useless for predicting the immediate next token. They identify three mechanisms through which features emerge during training: direct learning (features useful for the next token), pre-caching (features computed at one position to be useful at later positions), and circuit sharing (features learned because the same parameters serve different functions at different positions). Features driven by pre-caching and circuit sharing are what the authors call "next-token-prediction-useless" (NTP-useless) at the position where they are computed: removing them does not degrade the model's prediction at that token, yet they serve the model's internal computation downstream. In experiments on Gemma 2, features with high pre-caching influence are concentrated in formal reasoning domains (code, logical structure), confirming that the model's deepest structured representations are not visible in its token output and are oriented toward internal computational needs rather than surface-level prediction.

From this perspective, the long-standing debate over whether LLMs "truly reason" may be partly misplaced. The reasoning is present, but it is not where the field was looking. It lives in the latent space, in the dynamic composition of distributed circuits, in features that serve internal computation rather than token output. Natural language is an interface layer through which deeper computational processes are expressed, not the substrate of reasoning itself.

5. Reframing Emergence

The field's default explanation for why larger models suddenly acquire new capabilities is "emergence," a term borrowed from complex systems theory but used loosely to mean "capabilities that appear unpredictably at scale" (Wei et al., 2022). However, the preceding sections do indicate an underlying mechanism: the compression objective that produced the sentiment neuron at small scale, and NTP-useless features at large scale, does not simply accumulate representations as models grow. It constructs progressively richer ones. That insight is also not new. It belongs to a lineage that predates large language models by decades. Shannon (1948) established information theory as the mathematical framework for measuring the information content of a signal, defining entropy as the fundamental limit on lossless compression and showing that efficient communication requires exploiting the statistical structure of the source. Kolmogorov (1965) extended this to algorithmic complexity, establishing the minimum description length of a string as a foundational, computation-theoretic measure of its information content. Schmidhuber (1991; 2009) built on this to argue that compression progress, the discovery of novel regularities that shorten the description of observed data, is the fundamental drive behind curiosity, creativity, and learning. In this view, a system that compresses better understands more: intelligence is the rate at which an agent discovers more compact representations of its experience. Ma et al. (2022) formalised this as the twin principles of parsimony and self-consistency, and Ruffini (2026) proved the formal necessity via algorithmic information theory (discussed in the companion essays).

Earlier accounts of emergence in large language models attributed sudden improvements in reasoning ability to scale alone (Wei et al., 2022). The compression lineage suggests an alternate, direct interpretation: scale enables the formation of a richer space of latent operators, composable transformations that can be activated and recombined under context. What looks like "emergence" in these systems is the compression function finding richer regularities as the volume of data, computation depth and the capacity of the model increase. A discriminating case is instructive: if scale alone were sufficient, a larger model with weaker mathematical structure should always outperform a smaller one with stronger structure. The architectural evidence reviewed in §6 shows the opposite. CliffordNet achieves higher accuracy with 1.4M parameters than ResNet-18 does with 11.2M, precisely because its algebraic structure compresses the relevant geometric relationships more efficiently. Scale without the right compression structure does not produce the same capabilities; it is the quality of the compression, not merely its quantity, that determines what operators emerge.

The sentiment neuron was a single operator (a sentiment detector) emerging from compression at modest scale. At larger scale, thousands of such operators emerge, encoding reusable transformations, patterns, and invariants derived from training data. Domains such as mathematics, which are inherently structured and transformation-driven, align particularly well with this capability, allowing models to generate outputs that appear novel or insightful (Cobbe et al., 2021; Lewkowycz et al., 2022). These outputs are not the result of explicit symbolic manipulation but of structured generalisation within a learned operator space.

In-context learning (ICL) is the mechanism through which these operators are deployed. Rather than simply conditioning on prior tokens, the model infers a task from the prompt and adapts its internal computation accordingly, functioning as a form of meta-learning (Akyurek et al., 2022; von Oswald et al., 2023). This induces a fast inner loop that selects and configures relevant computational pathways within the latent space. The adaptation occurs below the level of tokens: what we observe in the output is a downstream projection of the latent computation, not the computation itself.

6. Architecture as Compression: The Relationship Between Mathematics and Engineering

The compression lineage traced above (Shannon → Kolmogorov → Schmidhuber → Ma et al.) applies not only to what models learn but to how they are built. If compression is the fundamental principle, then architectures grounded in the right mathematics should outperform those assembled from engineering heuristics, because the mathematics already formalises the structural relationships that heuristics approximate piecemeal. Recent work provides evidence for this claim at three levels of architectural design.

Algebraic completeness replaces modular complexity. For years, deep learning architectures have been dominated by increasingly complex stacks of engineered modules: convolutional blocks, attention layers, channel mixers, feed-forward networks, residual pathways, and normalisation schemes. Each component addresses one aspect of the representation problem, and each new combination is announced as an architectural advance. He et al. (2016) introduced residual connections and were widely celebrated for "solving" deep network training, but the mechanism was an engineering patch for gradient flow, not a principled account of how information should aggregate across depth. Ji (2026) challenges this entire design philosophy with CliffordNet, a vision backbone built around a single algebraic operation: the Clifford geometric product uv = u·v + u∧v, which simultaneously captures feature coherence (via the generalised inner product) and structural variation (via the exterior product). The mathematical structure of Clifford Algebra, developed in the nineteenth century by William Kingdon Clifford as a unification of Hamilton's quaternions and Grassmann's algebra of extensions, already formalises the interaction mechanisms that conventional architectures distribute across separate modules. Standard dot-product attention compresses the relationship between two vectors onto a scalar, discarding the directional and structural information (the bivector component) inherent in the feature space. The result is that feed-forward networks, previously considered indispensable for channel mixing in vision transformers, become redundant when geometric interactions are modelled completely. CliffordNet-Nano achieves 77.82% accuracy on CIFAR-100 with 1.4M parameters and strict O(N) complexity, outperforming ResNet-18 (76.75%, 11.2M parameters) with roughly eight times fewer parameters. The deeper variant, CliffordNet-64 (8.6M parameters), surpasses both ResNet-50 and DenseNet-121 without any feed-forward layers. The architectural implication is pointed: when the algebra is complete, half the architecture disappears.

Learnable functions replace fixed activations. Liu et al. (2025) introduce Kolmogorov-Arnold Networks (KANs), grounded in the Kolmogorov-Arnold representation theorem, which establishes that any continuous multivariate function can be decomposed into compositions of univariate functions and addition. Where standard multi-layer perceptrons place fixed activation functions on nodes, KANs place learnable activation functions (parametrised as splines) on edges, replacing every weight parameter with a univariate function. The architectural consequence mirrors CliffordNet's: much smaller KANs achieve comparable or better accuracy than much larger MLPs, with faster neural scaling laws and substantially greater interpretability. The theorem, proved in the 1950s, already specifies the decomposition that MLPs approximate with large parameter counts and brute-force connectivity. As Liu observes, MLPs have simple atoms but complicated ways to combine them; KANs have expressive atoms but simple combinatorics. The mathematical structure does more work, so the engineering scaffolding can do less.

Structural duality replaces fixed accumulation. The Kimi team (2026) demonstrate a third instance of mathematical formalisation completing an engineering convention. Standard residual connections, the backbone of modern transformers since He et al. (2016), accumulate all layer outputs with fixed unit weights, a design choice motivated by gradient-flow considerations rather than information-theoretic principles. Attention Residuals (AttnRes) recognise a formal duality between depth-wise accumulation in deep networks and sequential recurrence in RNNs: both compress prior information into a single state through a fixed additive rule. Just as self-attention replaced recurrence for sequence modelling by enabling selective retrieval, AttnRes replaces fixed residual accumulation with learned softmax attention over depth, allowing each layer to selectively aggregate information from all preceding layers. The mathematical insight is that standard residuals are a degenerate case of depth-wise linear attention; AttnRes generalises to the full softmax case with negligible overhead. Scaling law experiments show that AttnRes consistently outperforms standard residuals across model sizes, with Block AttnRes matching a baseline trained with 1.25× more compute. Integrated into the 48B-parameter Kimi Linear architecture and pre-trained on 1.4 trillion tokens, AttnRes improves performance across all downstream benchmarks, with the largest gains in multi-step reasoning (GPQA-Diamond +7.5) and code generation (HumanEval +3.1), precisely the tasks that benefit most from improved depth-wise information flow.

Completion, not correction

These three results converge on a principle that the compression lineage predicts. In each case, the engineering heuristic was not arbitrary: it embodied a partial mathematical intuition (gradient flow as identity mapping, dot products as similarity, additive accumulation as a simple prior over depth). What the formalisation achieves is not correction but completion: making the implicit geometric structure explicit, provable, and composable. CliffordNet completes the geometric structure that dot-product attention was already partially enacting, restoring the bivector component that the scalar projection had discarded. KANs replace the brute-force connectivity of MLPs with the decomposition that the Kolmogorov-Arnold theorem already guaranteed. AttnRes generalises the fixed additive rule of residual connections to the selective case that linear algebra always permitted. In each case, much of the engineering scaffolding becomes redundant once the mathematics is made complete.

A caveat is necessary: mathematical grounding is necessary but not sufficient. Not every mathematically motivated architecture succeeds. Capsule networks (Sabour, Frosst, and Hinton, 2017) were grounded in a principled geometric account of part-whole relationships, yet they have not scaled competitively to large vision benchmarks, in part because the routing-by-agreement mechanism introduces optimisation difficulties that the theory did not anticipate. The claim here is not that any formalisation will outperform any heuristic, but that when the formalisation captures the right structural invariants of the domain, the resulting architecture compresses more efficiently and the engineering scaffolding built around the missing structure becomes redundant.

Training geometry and data geometry

The principle extends beyond architecture to the training objective itself. Autoregressive (AR) next-token prediction factorises the joint distribution left-to-right: p(x) = ∏ p(xᵢ | x₍<ᵢ₎). The chain rule guarantees exactness, but the factorisation imposes a strong inductive bias: global structure must be reconstructed through a chain of unidirectional conditionals. For language, where communication unfolds temporally, this bias at least corresponds to a real process. For domains whose data manifolds are intrinsically non-sequential (protein structure, gene regulatory networks, chromatin topology), the mismatch is fundamental: a protein's spatial contacts have no preferred linear ordering, yet AR training requires each residue to be predicted as a conditional on its prefix. Masked and diffusion-based training objectives relax this constraint, modelling the joint distribution through iterative bidirectional refinement across all positions simultaneously. The transformer's forward-pass machinery is already order-agnostic; what changes is the learning signal that sculpts its latent geometry. Bidirectional protein language models (Lin et al., 2023) and diffusion-based structure generators (Watson et al., 2023) already demonstrate that non-sequential training develops representations better matched to non-sequential domains. The AR objective, in other words, is itself an engineering convention (historically dominant, computationally convenient, but not mathematically necessary), and relaxing it follows the same logic as the architectural cases above: when the training signal's geometry aligns with the data manifold's geometry, the model compresses more efficiently.

The principle extends further still, from the training objective to the data that enters it. Even within a fixed architecture and representation space, the geometric profile of which data is selected for learning shapes the trajectory of compression. Thummala and Abdelrazek (2026) demonstrate this in active learning: using intrinsic dimensionality (ID) — the effective degrees of freedom of data in a representation space — as a geometric diagnostic, they show that different acquisition strategies induce measurably different geometric regimes. Uncertainty-based strategies concentrate labelling budget on geometrically complex, high-ID regions; coverage-based strategies anchor in structurally coherent, low-ID regions first and expand outward gradually. The critical finding is that this ordering is not neutral. Controlled experiments show that prioritising low-ID samples early yields substantially better learning, while allocating budget to high-ID regions prematurely leads to inefficient learning — despite those samples being individually valid. Two strategies can reach similar accuracy through geometrically different paths, a difference invisible to accuracy curves but legible through intrinsic dimension. The result is a data-level instance of Schmidhuber's compression progress principle (§5): just as compression progress depends on the sequential discovery of regularities — simpler structures first, then the complex patterns that presuppose them — the geometry of what the system learns from has a natural curriculum. The system must consolidate structurally coherent, low-dimensional regularities before it can productively absorb high-dimensional complexity. And just as scale without the right compression structure does not produce the same capabilities (§5), data without the right geometric ordering does not produce the same learning efficiency. The geometry of the signal matters: not only what data is available, but the structural properties of what is selected and the order in which it enters the compression process.

What mathematics contributes

The implications for the broader argument of this essay are direct. If the locus of machine intelligence is in the latent operator space (§5), then the quality of that space depends not on the number of architectural modules stacked upon it but on the mathematical coherence of the operations that construct it. The compression objective does not merely select for useful representations of the world; it selects, over the trajectory of research, for mathematical structures that are themselves more powerful descriptions of the computational operations the model needs.

This pattern invites reflection on what mathematics actually contributes. Mathematics does not merely compress known regularities into shorter descriptions, though it does that (Shannon, Kolmogorov). It also creates new symbolic vocabulary (quaternions, geometric algebras, representation theorems) that makes structural relationships expressible for the first time, and it formalises phenomena into symbolic structures whose internal consistency (axioms, proofs, lemmas, geometric reasoning) allows them to be manipulated, composed, and extended with guaranteed reliability. These are distinct contributions: compression reduces, formalisation translates into a manipulable symbolic form, and conceptual creation extends what can be represented at all.

Wigner (1960) famously called the correspondence between mathematical structures and physical reality "unreasonable." The architectural evidence reviewed above does not dissolve this puzzle entirely, but it narrows it for one important class of cases. In engineered systems whose design space is constrained by known structural relationships (geometric interactions, function decomposition, depth-wise information flow), the effectiveness of mathematics is not mysterious: the formalisation captures the same invariants that govern the system's behaviour, and architectures built around those invariants compress more efficiently because they are working with the structure rather than around it. Whether this account extends to Wigner's broader concern — the "unreasonable" fit between abstract mathematics and physical reality in domains where no engineering constraint guided the discovery — remains an open question that the present evidence does not settle.

The porous boundary between mathematics and engineering

The relationship between mathematics and engineering is, in practice, more porous than either community's narrative suggests. The cases above are not stories of blind engineering corrected by pure mathematics. Researchers carry mathematical intuitions into their engineering: He et al.'s residual connections embody a geometric intuition about identity mappings and gradient flow; dot-product attention draws on ideas from associative memory and kernel methods; even the standard MLP reflects an implicit (if incomplete) grasp of function approximation. These heuristics are not arbitrary; they are partial mathematical insights that have not yet been fully formalised. What CliffordNet, KAN, and AttnRes achieve is completion rather than correction: making the implicit geometry explicit, provable, and composable.

The reverse direction is equally real but operates differently. Engineering regularly produces artefacts whose behaviour existing mathematical vocabulary has not yet made expressible, generating phenomena that demand new formalisation. The mechanistic interpretability programme reviewed in §3 is precisely this: Elhage et al.'s circuit decompositions, Olsson et al.'s induction heads, and von Oswald et al.'s characterisation of in-context learning as implicit gradient descent are all new symbolic vocabularies created to make legible what engineers had already built. Neural tangent kernel theory (Jacot, Gabriel, and Hongler, 2018) and the double-descent phenomenon (Belkin et al., 2019) are further cases where engineering practice generated phenomena that classical theory lacked the vocabulary to express, and new mathematics was developed to formalise them.

The field advances not through a clean handoff between two separate communities but through researchers moving between geometric intuition, engineering pragmatism, and formal proof: engineering explores and builds (often guided by partial mathematical awareness), mathematics formalises what engineering discovers into internally consistent frameworks that both complete and extend the vocabulary of the field, and those frameworks reveal structure that sharpens the next generation of intuitions. Architecture design, viewed through this lens, is itself a formalisation process — but the word "formalisation" operates at two distinct levels that should not be conflated. At the model level, formalisation means the compression of training data into structured latent representations (the process traced in §§1-5). At the research level, formalisation means the translation of engineering design knowledge into explicit mathematical frameworks with internal consistency guarantees (the process traced in this section). The two levels interact — better mathematical frameworks at the research level produce architectures whose model-level compression is more efficient — but they are not the same process, and conflating them obscures the causal direction: it is the research-level formalisation that enables more powerful model-level compression, not the reverse.

7. The Substrate and the Interface

The evidence reviewed in §§1-6 spans vision models (CliffordNet), function approximation (KANs), deep network design (AttnRes), protein structure prediction, and language modelling. The compression principle and the substrate/interface distinction that emerge from this evidence are not specific to any one architecture or modality. They suggest a general layered view of trained intelligent systems:

Data space: raw inputs, observations, and context
Latent operator space: internal representations encoding transformations and possibilities
Interface space: language, APIs, or other modalities through which the system communicates
Action space: decisions, interventions, and real-world effects

These layers are not equally accessible. The data space and latent operator space are primarily shaped during training: the compression objective, the training data, and the architectural choices examined in §6 collectively determine what operators the system can construct and what structural relationships it can represent. Once training is complete, these substrate layers are largely fixed. Current engineering effort concentrates overwhelmingly on the interface and action layers. In language modelling, this takes the form of prompt design, retrieval augmentation, tool routing, and output parsing. In vision, it appears as data augmentation pipelines and task-specific heads bolted onto frozen backbones. In protein design, it manifests as post-hoc filtering and scoring of generated candidates. The pattern is general: practitioners work at the surface because the substrate is expensive to modify and opaque to inspect.

The asymmetry reveals a structural tension. The evidence reviewed here establishes that the richest representations, the ones that encode world models, support latent reasoning, and develop features oriented toward internal computation rather than surface prediction, live in the latent operator space. Yet this is precisely the layer over which practitioners have the least direct control at deployment. The interface layer, where most optimisation occurs, is the shallowest projection of that deeper structure. Interface-level interventions (prompting strategies in language models, augmentation pipelines in vision, scoring functions in generative biology) can select among the system's existing capabilities but cannot extend them. Approaches such as tool use, external planning modules, and structured reasoning frameworks begin to address this gap by coupling latent capabilities to action spaces that extend effective reach. But the deeper opportunity, suggested by §6's architectural evidence, is that gains at the substrate level propagate upward through all layers in ways that interface-level interventions cannot replicate. CliffordNet's 8× parameter reduction, KAN's faster neural scaling laws, and AttnRes's 1.25× compute advantage are all consequences of changing the mathematical structure of the substrate. No amount of interface-level engineering can recover what a poorly structured substrate never learned to represent.

The practical implication is that trained systems, across modalities, do considerably more than produce outputs matched to their training objective. They construct structured internal representations through which data is compressed into operators that support reasoning, generalisation, and action. Large language models are currently the most prominent instance, because the richness of human language as a training signal produces an unusually deep and versatile operator space. But the principle is general: any system whose compression objective encounters sufficient structural regularity in its training data will develop latent operators that go beyond what the objective nominally requires (as the NTP-useless features of §4 demonstrate). Recognising the primacy of the substrate also clarifies where current systems fall short. Their limitations, including the absence of autonomous learning (Dupoux, LeCun, and Malik, 2026), weak action coupling to domains whose structural relationships have not yet been formalised into frameworks accessible to the training process, and a static abstraction layer that does not reorganise through environmental interaction, are substrate-level constraints that no interface-level intervention can fully compensate for.

The substrate is richer than we access

Yet the same framework suggests where the immediate opportunity lies. The characterisation of the substrate as "largely fixed" after training understates what is already happening at inference time. In-context learning (ICL), as §3 established, functions as a form of implicit gradient descent within the forward pass (von Oswald et al., 2023): the model adapts its internal computation to the task at hand without any parameter update. The substrate is not inert after training; it is a computational engine that reconfigures itself in response to context. Recent theoretical work confirms that this engine is more powerful than the "statistical pattern matching" account implies. Li and Wang (2025) proved that constant bit-size transformers are Turing complete, requiring only sufficient context length, not increased precision or parameters, to simulate any bounded Turing machine. Their follow-up (Li and Wang, 2025b) showed this simulation can be made efficient, with optimal context windows and minimal chain-of-thought overhead. Tzamos et al. (2026) turned the theory into a working demonstration: a standard transformer with hand-crafted weights that embeds a full WebAssembly virtual machine, generating and executing arbitrary programs entirely within the forward pass, no external interpreter, no tool calls. The model solves problems (including hard Sudoku instances requiring millions of computational steps) by running a real solver natively inside its own inference loop. An important caveat: the Percepta weights were hand-crafted by researchers, not discovered by gradient descent during training. The result is therefore best read as a ceiling proof — it establishes that the transformer architecture is capable of hosting arbitrary computation, not that current training procedures will find such configurations. Whether gradient-based training can converge on weight configurations with comparable computational richness remains an open empirical question, though the Turing completeness proofs (Li and Wang, 2025; 2025b) suggest the theoretical capacity is present in the architecture itself.

These results are existence proofs, not characterisations of typical behaviour: they establish in principle that the transformer substrate can support computational processes far richer than token prediction, without claiming that any particular trained model routinely exploits this capacity. Nonetheless, the practical implication is that the gap between the substrate's theoretical capabilities and its currently accessible capabilities is likely much larger than current systems exploit. Agentic frameworks (code execution environments, tool-use loops, interactive assistants) expand the action coupling and, in doing so, open new signal pathways through which the model's existing latent operators can engage with consequences, iterate on feedback, and exploit competences that were present but unreachable under constrained output modalities. Each new action pathway reveals a slice of latent competence the substrate already possessed. The substrate-level improvements argued for in §6 remain the highest-leverage long-term investment, but the near-term implication is equally clear: the returns to expanding action coupling on the current substrate are far from exhausted.

8. Connection to the Essay Series

The companion essays advance two claims that the computational evidence reviewed here can now ground empirically.

The first is that an agent is formally characterised by its compression function: the mapping from unbounded history to finite, actionable representation ("The Agent, Not the Task"). The evidence confirms this directly. Transformers trained on next-token prediction build compressed world models (§2), implement dynamic meta-learning through latent adaptation (§3), and develop internal features oriented toward computation rather than surface prediction (§4). The quality of the agent is the quality of this compression. The architectural evidence of §6 extends the point: the parsimony principle (Ma et al., 2022) applies not only to what the system represents but to how the system is built. The best compressor is not the one with the most components but the one whose components are mathematically complete.

The second is that intelligence is bounded by a three-part constraint: the signal source (what data is available for compression), the abstraction layer (how that data can be represented and transformed), and the action coupling (whether representations connect to interventions in the world). This framework, derived in "The Mapmaker's Alphabet" from four converging definitions of intelligence (Chollet, 2019; Levin, 2019; Ma et al., 2022; Bennett, 2024), predicts exactly the pattern §7 identifies. The abstraction layer is richer than the "statistical parrot" account suggests. But it operates within the signal source constraint (what reaches the latent space is still determined by training data), and the substrate-level limitations of current systems (no autonomous learning, weak action coupling, static abstraction) map precisely onto the three gaps the constraint predicts.

The broader implication is that language is one surface among several through which deeper computational processes are expressed. Understanding and engineering these systems requires engaging directly with the latent structures that give rise to the surface, not optimising the surface itself.

Summary

AI systems trained to predict text build internal models of the world that are far richer than the text they produce. The evidence from accidental feature discovery through mechanistic circuit analysis to hidden reasoning in continuous space consistently points to the same conclusion: the interesting part of these systems is not what they say, but what they silently learn in order to say it.

What determines the quality of that hidden knowledge is not primarily dependant on how big the model is. It is how well the architecture's mathematics matches the structure of the problem. When the mathematics is right, smaller systems routinely outperform much larger ones. When it is wrong, no amount of scale compensates. The relationship between mathematical insight and engineering intuition is not one of correction but of completion & it runs in both directions. And the principle reaches beyond the architecture: the geometric properties of the data selected for learning, and the order in which it enters the compression process, shape what the system can learn as fundamentally as the mathematics that processes it. Structure is the organising principle at every level — from the algebra of the model, to the geometry of the signal it consumes, to the quality of the representations it constructs.

The practical implications split into two time horizons. Over the long term, the highest-leverage investment is in the mathematical foundations: the compression architecture that determines what a system can represent at all. But the near-term opportunity is equally real and more immediately actionable. These systems already know more than they can show. Every new way of connecting their internal knowledge to the world (through tools, code execution, feedback loops, agentic frameworks) reveals capabilities that were already present but had no way out. The systems are not waiting for better architectures to become useful. They are waiting for better bridges to what they already know.

References

Akyurek, A.F., Schuurmans, D., Andreas, J., Ma, T. and Zhou, D. (2022) 'What learning algorithm is in-context learning? Investigations with linear models', arXiv preprint, arXiv:2211.15661.

Bennett, M.T. (2024) 'Are biological systems more intelligent than artificial intelligence?', arXiv preprint, arXiv:2405.02325.

Bhambri, S., Biswas, U. and Kambhampati, S. (2025) 'Interpretable traces, unexpected outcomes: Investigating the disconnect in trace-based knowledge distillation', arXiv preprint, arXiv:2505.13792.

Belkin, M., Hsu, D., Ma, S. and Mandal, S. (2019) 'Reconciling modern machine-learning practice and the classical bias-variance trade-off', Proceedings of the National Academy of Sciences, 116(32), pp. 15849-15854. doi:10.1073/pnas.1903070116.

Chollet, F. (2019) 'On the measure of intelligence', arXiv preprint, arXiv:1911.01547.

Cobbe, K., Kosaraju, V., Bavarian, M., Chen, M., Jun, H., Kaiser, L., Plappert, M., Tworek, J., Hilton, J., Nakano, R., Hesse, C. and Schulman, J. (2021) 'Training verifiers to solve math word problems', arXiv preprint, arXiv:2110.14168.

Dupoux, E., LeCun, Y. and Malik, J. (2026) 'Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive science', arXiv preprint, arXiv:2603.15381.

Elhage, N., Nanda, N., Olsson, C., Henighan, T., Joseph, N., Mann, B., Askell, A., Bai, Y., Chen, A., Conerly, T., DasSarma, N., Drain, D., Ganguli, D., Hatfield-Dodds, Z., Hernandez, D., Jones, A., Kernion, J., Lovitt, L., Ndousse, K., Amodei, D., Brown, T., Clark, J., Kaplan, J., McCandlish, S. and Olah, C. (2021) 'A mathematical framework for transformer circuits', Transformer Circuits Thread.

Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J. and Tian, Y. (2025) 'Training large language models to reason in a continuous latent space', Conference on Language Modelling (COLM), arXiv:2412.06769.

He, K., Zhang, X., Ren, S. and Sun, J. (2016) 'Deep residual learning for image recognition', Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778.

Jacot, A., Gabriel, F. and Hongler, C. (2018) 'Neural tangent kernel: Convergence and generalization in neural networks', Advances in Neural Information Processing Systems 31 (NeurIPS), pp. 8571-8580.

Ji, Z. (2026) 'CliffordNet: All you need is geometric algebra', arXiv preprint, arXiv:2601.06793.

Kambhampati, S., Valmeekam, K., Guan, L., Verma, M., Stechly, K., Bhambri, S., Saldyt, L.P. and Murthy, A.B. (2024) 'Position: LLMs can't plan, but can help planning in LLM-Modulo frameworks', Proceedings of the 41st International Conference on Machine Learning (ICML), arXiv:2402.01817.

Kambhampati, S., Stechly, K., Valmeekam, K., Saldyt, L.P., Bhambri, S. and Pal, S. (2025) 'Position: Stop anthropomorphizing intermediate tokens as reasoning/thinking traces!', arXiv preprint, arXiv:2504.09762.

Kimi Team (2026) 'Attention Residuals', Technical Report, arXiv:2603.15031. Available at: https://github.com/MoonshotAI/Attention-Residuals.

Kolmogorov, A.N. (1957) 'On the representation of continuous functions of many variables by superposition of continuous functions of one variable and addition', Doklady Akademii Nauk SSSR, 114(5), pp. 953-956.

Kolmogorov, A.N. (1965) 'Three approaches to the quantitative definition of information', Problems of Information Transmission, 1(1), pp. 1-7.

Levin, M. (2019) 'The computational boundary of a "Self": Developmental bioelectricity drives multicellularity and scale-free cognition', Frontiers in Psychology, 10, 2688. doi:10.3389/fpsyg.2019.02688.

Lewkowycz, A., Andreassen, A., Dohan, D., Dyer, E., Michalewski, H., Ramasesh, V., Slone, A., Anil, C., Schlag, I., Gutman-Solo, T., Wu, Y., Neyshabur, B., Gur-Ari, G. and Misra, V. (2022) 'Solving quantitative reasoning problems with language models', Advances in Neural Information Processing Systems 35 (NeurIPS).

Li, K., Hopkins, A.K., Bau, D., Viegas, F., Pfister, H. and Wattenberg, M. (2023) 'Emergent world representations: Exploring a sequence model trained on a synthetic task', Proceedings of the 11th International Conference on Learning Representations (ICLR 2023), arXiv:2210.13382.

Li, K., Hopkins, A.K., Bau, D., Viegas, F., Pfister, H. and Wattenberg, M. (2023b) 'Othello-GPT has a linear emergent world model', ICML 2023 Workshop on Deployment Challenges for Generative AI.

Li, Q. and Wang, Y. (2025) 'Constant bit-size transformers are Turing complete', Advances in Neural Information Processing Systems 38 (NeurIPS), arXiv:2506.12027.

Li, Q. and Wang, Y. (2025b) 'Efficient Turing machine simulation with transformers', arXiv preprint, arXiv:2512.00003.

Lin, Z., Akin, H., Rao, R., Hie, B., Zhu, Z., Lu, W., Smelov, N., Verkuil, R., Kabeli, O., Shmueli, Y., dos Santos Costa, A., Fazel-Zarandi, M., Sercu, T., Candido, S. and Rives, A. (2023) 'Evolutionary-scale prediction of atomic-level protein structure with a language model', Science, 379(6637), pp. 1123-1130. doi:10.1126/science.ade2574.

Liu, Z., Wang, Y., Vaidya, S., Ruehle, F., Halverson, J., Soljačić, M., Hou, T.Y. and Tegmark, M. (2025) 'KAN: Kolmogorov-Arnold Networks', Proceedings of the 13th International Conference on Learning Representations (ICLR 2025), arXiv:2404.19756.

Ma, Y., Tsao, D. and Shum, H.-Y. (2022) 'On the principles of parsimony and self-consistency for the emergence of intelligence', Frontiers of Information Technology & Electronic Engineering, 23(9), pp. 1298-1323. doi:10.1631/FITEE.2200297.

Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B. and Olah, C. (2022) 'In-context learning and induction heads', Transformer Circuits Thread, arXiv:2209.11895.

Radford, A., Jozefowicz, R. and Sutskever, I. (2017) 'Learning to generate reviews and discovering sentiment', arXiv preprint, arXiv:1704.01444.

Radford, A., Narasimhan, K., Salimans, T. and Sutskever, I. (2018) 'Improving language understanding by generative pre-training', OpenAI Technical Report.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D. and Sutskever, I. (2019) 'Language models are unsupervised multitask learners', OpenAI Technical Report.

Rofin, M., Naghiyev, J. and Hahn, M. (2026) 'Understanding the emergence of seemingly useless features in next-token predictors', Proceedings of the 14th International Conference on Learning Representations (ICLR 2026), arXiv:2603.14087.

Ruffini, G. (2026) 'The algorithmic regulator', Entropy, 28(3), 257. doi:10.3390/e28030257.

Sabour, S., Frosst, N. and Hinton, G.E. (2017) 'Dynamic routing between capsules', Advances in Neural Information Processing Systems 30 (NeurIPS), pp. 3856-3866.

Schmidhuber, J. (1991) 'Curious model-building control systems', Proceedings of the IEEE International Joint Conference on Neural Networks (IJCNN), vol. 2, pp. 1458-1463.

Schmidhuber, J. (2009) 'Simple algorithmic theory of subjective beauty, novelty, surprise, interestingness, attention, curiosity, creativity, art, science, music, jokes', Journal of the Society of Instrument and Control Engineers, 48(1), pp. 21-32. arXiv:0812.4360.

Shannon, C.E. (1948) 'A mathematical theory of communication', The Bell System Technical Journal, 27(3), pp. 379-423. doi:10.1002/j.1538-7305.1948.tb01338.x.

Thummala, P. and Abdelrazek, M. (2026) 'Intrinsic dimension dynamics in active learning: A geometric diagnostic of acquisition behavior', GRaM Workshop at ICLR 2026 (Tiny Paper Track).

Tzamos, C. et al. (2026) 'Can LLMs be computers?', Percepta AI Technical Blog, March 2026. Available at: https://www.percepta.ai/blog/can-llms-be-computers.

von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A. and Vladymyrov, M. (2023) 'Transformers learn in-context by gradient descent', Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202, pp. 35151-35174.

Watson, J.L., Juergens, D., Bennett, N.R., Trippe, B.L., Yim, J., Eisenach, H.E., Ahern, W., Borber, A.J., Ragotte, R.J., Milles, L.F., Wicky, B.I.M., Hanikel, N., Pellock, S.J., Courbet, A., Sheffler, W., Wang, J., Venkatesh, P., Sappington, I., Torres, S.V., Laber, A., Mulligan, V.K., Fiber, R., Vecchio, K.D., De Moura, M., Brunette, T.J., Rodionov, D., Yeh, A.H., Bera, A.K., Kang, A. et al. (2023) 'De novo design of protein structure and function with RFdiffusion', Nature, 620(7976), pp. 1089-1100. doi:10.1038/s41586-023-06415-8.

Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J. and Fedus, W. (2022) 'Emergent abilities of large language models', Transactions on Machine Learning Research.

Wigner, E.P. (1960) 'The unreasonable effectiveness of mathematics in the natural sciences', Communications in Pure and Applied Mathematics, 13(1), pp. 1-14. doi:10.1002/cpa.3160130102.