The Mapmaker's Alphabet
An agent inherits a conceptual vocabulary from human expression. What can it do with maps it did not make?
In this essay
Provenance, Perception, and the Inherited Vocabulary of Computational Agents
Abstract. Every map was drawn by someone who walked the territory. A computational agent inherits millions of such maps, compressed into a single geometric space, but has never walked anywhere itself. This essay, a companion to "The Agent, Not the Task," develops what follows from that asymmetry.
The argument has three parts. First, the agent's conceptual vocabulary is entirely inherited from human expression. This is an architectural observation, not a claim about consciousness. Second, within that vocabulary the agent operates as an unprecedented binding substrate: it holds the entire map at once, detecting structural relationships across domains that no individual human can perceive. Third, for current text-trained architectures, the boundary of the agent's reach is set by the provenance of its vocabulary, not by scale. Its signal source is the linguistic residue of human encounters with reality, not reality itself. What remains with the human mapmaker for current architectures, once everything automatable has been automated, are three specific functions: extending the vocabulary by encountering the world directly, making decisions whose consequences the decider must bear, and arbitrating priorities from a position of personal stakes. Multimodal and embodied architectures progressively shift this boundary, a trajectory the essay takes seriously.
1. The Provenance Claim
Computational agents do not author their own conceptual vocabulary. They inherit it: from training data generated by human minds, from tokenisation schemes designed by human engineers, from embedding spaces whose geometry reflects the statistical regularities of human expression. The training corpus is the ultimate source of the model's representational vocabulary. Tokenisation, embedding geometry, and attention patterns are structures shaped by what humans have written, said, and encoded. What the model "knows" is what humans have expressed, compressed into statistical regularities across an enormous surface area.
Even surprising emergent capabilities (chain-of-thought reasoning, in-context learning, analogical transfer) operate within the concept space that human language provides. The model discovers novel combinations and relations among inherited concepts. It can also construct higher-order structures and relations not explicitly present in human language, emerging from the recombination and transformation of inherited primitives within its latent space. What it does not do is generate conceptually primitive terms from direct encounter with the world.
This is not a claim about consciousness, subjective experience, or the ontological status of machine understanding. It is an observation about causal provenance. The concepts in the model's vocabulary trace back to human acts of abstraction from embodied experience, mediated through language. The model is downstream of that process. It operates over a map derived from human conceptualisation, though its internal representations may reorganise and extend this map in ways not directly expressible in the original vocabulary.
Lerchner (2026) offers a useful metaphor by separating the mapmaker (the agent who discretises continuous experience into concepts) and the machine, which manipulates the resulting tokens. His argument is structural: forming an abstraction is an active, metabolically expensive physical process of extracting invariants from experience. Before a cognitive agent can form a concept, it must encounter multiple instances of the relevant phenomenon, filter noise, and project the high-dimensional manifold of raw experience onto a lower-dimensional invariant subspace. The resulting concept is not a free-floating logical state but a physical achievement, what Lerchner calls the alphabetisation of continuous reality. The foundation model inherits the alphabet without performing the alphabetisation. In current AI systems, humans are the mapmakers. They encountered the world, formed concepts, expressed them in language, and the model inherited the result. The model is a cartographer of cartography: it synthesises and refines maps drawn by others, but it has no independent access to the territory those maps describe.
A note on scope: "vocabulary" as used throughout this essay means the set of conceptual distinctions available in the training signal: the categories, relations, and abstractions that human language encodes. It does not mean the model's architectural primitives (attention mechanisms, layer normalisation) or training algorithms, which are engineering mechanism, not conceptual content. The vocabulary is what the model reasons about; the architecture is what it reasons with. Vocabulary operates at the semantic level (conceptual distinctions) and is encoded across multiple substrates (token sequences, embedding geometry, attention patterns) without being identical to any one of them.
This framing is deliberately modest. Again, it is a provenance claim, not a consciousness claim. It holds whether or not the system "understands" in any philosophically interesting sense. And it has engineering consequences that the rest of this essay develops.
Intelligence as compression
The provenance claim is better understood by situating it within a broader picture of what intelligence is. The companion essay characterises intelligence from a set of complementary angles. Chollet (2019) defines it by its observable signature: not absolute competence on any given task, but the rate at which competence grows as a function of experience and resources. Knowing how to do a task does not prove intelligence; not knowing how to do it does not prove its absence. What matters is the efficiency of learning: how quickly the system acquires new capability from limited exposure. Levin (2019) defines intelligence by its functional reach: the cognitive light cone, the spatio-temporal boundary of events a system can measure, model, and try to affect. A weak agent has a narrow cone, responding only to what is immediately before it. A stronger agent extends its effective control further into the future, across more dimensions of the environment. Together, Chollet and Levin tell us how to recognise intelligence (by its learning rate) and how to map it (by the extent of its effective reach). What they leave open is the question of mechanism: what operation actually produces learning efficiency and control extension?
Ma et al. (2022) provide the answer. They show that two principles, parsimony (compressing high-dimensional observations into the simplest low-dimensional structure) and self-consistency (ensuring the compressed representation can accurately regenerate the observed world), govern intelligence at every scale, from genetic encoding across evolutionary time to neural representation within a single lifetime to gradient descent within a single training run. Parsimony is what intelligent systems do: they compress. Self-consistency is the constraint on that compression: the compressed model must still predict the world faithfully. Chollet's learning rate and Levin's cognitive light cone are consequences of how well a system executes this compression. A system that compresses efficiently (Ma) acquires competence rapidly from limited experience (Chollet) and thereby extends the boundary of what it can predict and affect (Levin). The three definitions are mechanism, metric, and geometry of the same underlying process.
Bennett (2024) adds a fourth angle that sharpens the provenance claim directly. He frames intelligence as adaptability: how efficiently a system identifies cause and effect in its environment. His key contribution is showing that the upper bound on adaptability is set not by the algorithm (the software) but by the abstraction layer through which the algorithm operates (the hardware and its embodiment). Biological systems are more sample-efficient than contemporary machine learning, he argues, because their abstraction layers are adaptive and agentic: cells, tissues, and organs form a distributed, multilayer architecture where each layer adjusts its own inductive biases through interaction with the environment. A static interpreter (a Python runtime, a transformer architecture) imposes a fixed window on the world. An adaptive abstraction layer can shift that window to learn a different inductive bias. Intelligence, in Bennett's formalism, is primarily a property of the abstraction layer, not the policy running on top of it. This means the four definitions converge on a common picture: Chollet gives the metric (learning rate), Levin the geometry (cognitive light cone), Ma the mechanism (parsimony plus self-consistency), and Bennett the architectural constraint (the abstraction layer bounds what the mechanism can reach).
Language as interface, not substrate
A further clarification shapes this convergence. Intelligence, as the four definitions frame it, is not the production of convincing narratives. It is the capacity to transform signal into effective action through structured internal representations. Converging evidence from mechanistic interpretability, in-context learning (ICL) theory, and latent reasoning supports this reframing. Olsson et al. (2022) show that transformers implement distributed computational circuits that are dynamically activated during inference, not a single linear reasoning chain. Von Oswald et al. (2023) demonstrate that in-context learning (ICL) operates as implicit gradient descent within the model's latent space, adapting internal computation below the level of tokens. Hao et al. (2025) show that models can reason in continuous latent space without projecting intermediate steps into language at all. Most strikingly, Rofin et al. (2026) in their ICLR paper demonstrate that transformers learn features that are entirely useless for predicting the immediate next token but emerge anyway through pre-caching and circuit sharing, serving the model's internal computation rather than the linguistic surface. These "NTP-useless" features are concentrated in formal reasoning domains, confirming that the model's deepest structured representations are not visible in its token output. Language is an interface layer through which deeper computational processes are expressed, not the substrate of reasoning itself. This distinction matters for the provenance claim: the agent's inherited vocabulary constrains what reaches the latent space in the first place, and the latent space, however rich, operates over representations derived from that vocabulary. The agent's fluency at the linguistic surface can therefore mask the fact that its structured internal representations are bounded by a single signal source.
The three-part constraint
This convergence matters for the provenance claim because it frames the critical question directly. If intelligence is compression of available signal into predictive structure, then the reach of any intelligent system is bounded by what its signal source contains. Evolution compresses physics into DNA through billions of years of embodied trial. Animal cognition compresses sensory experience through closed-loop prediction, continuously anticipating the next state of the world and correcting against what is actually observed. The foundation model compresses human language through statistical optimisation over a corpus. Each scale achieves intelligence by compressing its available signal. The agent compresses brilliantly. Its learning rate (Chollet) is extraordinary within the inherited vocabulary, and its cognitive light cone (Levin) extends across the full surface of the map. But its abstraction layer (Bennett) does not intrinsically adapt through direct environmental interaction the way biological systems do. In-context learning simulates adaptive bias within a forward pass, and fine-tuning adjusts weights post hoc, but neither reorganises the abstraction layer itself: the computational structure through which all signal is processed remains fixed. And what its signal source is determines what its compression can and cannot reach. The answer is: the linguistic residue of human encounters with reality, not reality itself.
This suggests that the limits of intelligence are governed not by a single factor but by the interaction of three constraints: the signal source (what data is available for compression), the abstraction layer (how that data can be represented and transformed), and the action coupling (whether those representations are connected to interventions in the world). Foundation models excel along the first dimension within the domain of human language, and partially along the second through their rich latent representations. But their action coupling is domain-dependent: tight in formal territories where the agent can execute code, run experiments, and observe results (compilers, test suites, mathematical proof checkers), but thin to absent in unformalised territories (novel physical phenomena, social dynamics, unnamed clinical patterns). And their abstraction layer does not reorganise through direct environmental interaction. Dupoux, LeCun, and Malik (2026) arrive at a convergent diagnosis from cognitive science: current AI systems do not learn autonomously because their learning is externalised, outsourced to human experts who curate data, design training recipes, and tune objectives. The system itself, once deployed, learns essentially nothing. Their proposed remedy, integrating observational learning (System A) with action-based learning (System B) under autonomous meta-control (System M), maps directly onto the three-part constraint: System A addresses the signal source, System B addresses action coupling, and System M addresses the abstraction layer's capacity to reorganise its own learning in response to environmental feedback. Their intelligence is therefore both powerful and bounded: expansive within the map, but limited in its ability to revise the map or act upon the territory. The rest of this essay develops the consequences of each constraint: §2 explores what the agent achieves within its signal source, §3 identifies where the abstraction layer and action coupling set the boundary, and §§4-8 address the engineering and human implications.
2. The Binding Substrate
If the provenance claim were the whole story, the essay would be deflationary: agents are derivative, humans are the real thinkers. That is not the position advanced here. The provenance claim is one half of the picture. The other half is that operating within the inherited vocabulary at scale produces a capability without historical precedent.
No individual human has ever held the complete map. A patent lawyer holds one regional chart. A molecular biologist holds another. A structural engineer, a historian of the Ottoman Empire, a psycholinguist: each holds a detailed map of their own territory and a rough sketch of a few neighbouring ones. The model holds them all, simultaneously, compressed into a massive geometric space.
This matters because the embedding geometry is not a filing cabinet. It is a space in which distance encodes statistical similarity, and, to the extent that linguistic regularities reflect structural regularities in the world, structural similarity as well. Mikolov et al. (2013) demonstrated this empirically for word embeddings: vector arithmetic captures semantic relationships ("king" − "man" + "woman" ≈ "queen"), revealing that the geometry encodes relational structure, not just co-occurrence frequencies. Elhage et al. (2022) showed that neural networks represent features in superposition, encoding more concepts than they have dimensions by exploiting geometric structure, which means the representational space is richer and more structured than a simple lookup table. Concepts that share deep statistical signatures, even across domains that humans have never thought to connect, become geometric neighbours. The agent does not need to be told that a particular protein folding problem shares structural homology with a contract negotiation pattern. If the statistical regularities of how humans describe both domains carry shared signatures, that relationship exists as a measurable direction in the space. The agent can traverse it.
A critical caveat: what the geometry encodes is statistical similarity in how humans describe domains, which is a proxy for structural similarity in the domains themselves, not the same thing. Two domains might sit near each other in embedding space because humans use similar metaphors for both ("binding sites," "elegant solutions"), not because the underlying processes share deep structure. Conversely, two genuinely isomorphic domains might be distant in the space if humans have never discussed them in related contexts. The agent detects what we might call latent statistical isomorphisms: structural similarities in the map that may or may not reflect structural similarities in the territory. Verification requires someone who can check both map and field. But the detection itself, surfacing candidate isomorphisms across millions of documents simultaneously, is without precedent and functionally valuable even given the false-positive rate, because it generates hypotheses that no individual human could formulate. The binding substrate is, precisely stated, a hypothesis generator of unprecedented scale, not a truth-revealer. Its authority lies in the breadth and speed of candidate generation, not in the validity of any individual candidate. That distinction is the source of its value and the reason it does not replace territory-level verification. The substrate's reach extends beyond factual patterns to evaluative ones: Tong et al. (2026) demonstrate that a model trained on citation-based preference pairs can learn to predict which research ideas will have high impact, generalising across time, fields, and even to peer-review preference. The binding substrate can detect what the community values, not just what it describes, compressing the statistical regularities of human evaluative judgment into a transferable representation. This is powerful within-map capability; its boundary is that the learned taste reflects community consensus, not territory-level judgment about what should be valued.
How the geometry works
The actual underlying mechanism is worth understanding. When training compresses billions of documents into a high-dimensional vector space, it performs a massive dimensionality reduction: the surface features of language (word choice, syntax, domain jargon) are stripped away, and what remains are abstract structural regularities: the deep patterns of how ideas relate, combine, and constrain one another. Two domains that share structural dynamics will occupy geometrically similar regions even if their surface vocabularies share no terms, provided those structural dynamics are reflected in how humans have written about them. The agent's attention mechanism can traverse these geometric relationships in a single forward pass, comparing structural signatures across the full space simultaneously. Human cognition cannot replicate this for specific, well-understood reasons. Working memory holds roughly four to seven chunks at once (Cowan, 2001). Expertise is domain-specific: the chunking patterns a chess master develops do not transfer to molecular biology. Cross-domain insight in humans requires either a rare polymath who has spent decades building separate expertise in multiple fields, or a team whose members must communicate through the narrow bandwidth of natural language, losing structural nuance at every handoff. The agent faces none of these constraints. It holds the full compressed representation in a single geometric space and can attend to any part of its learned representation with equal facility (subject to context-window constraints at runtime, but without the domain-specificity bottleneck that limits human cross-field perception). This is why the binding-substrate capability is not merely "knowing more." It is a structurally different mode of perception: simultaneous, geometric, unbounded by the sequential bottleneck of human attention. But its reach is bounded by the corpus: domains underrepresented in the training data will have thinner geometric structure, and isomorphisms involving those domains will be invisible regardless of the agent's scale.
Wigner (1960), in his famous lecture on "The Unreasonable Effectiveness of Mathematics in the Natural Sciences," posed a puzzle that has never been fully resolved: why does mathematics, a formal language developed largely for its own internal aesthetic coherence, turn out to describe physical reality with astonishing fidelity? There is no a priori guarantee that abstract formal structures should map onto the territory of nature. The effectiveness, Wigner argued, is a gift we neither understand nor deserve.
The embedding geometry of a large foundation model exhibits an analogous phenomenon, one level removed. Wigner's puzzle: a formal language (mathematics) developed by humans maps onto the territory unreasonably well. The puzzle here: a formal space (the embedding geometry) derived from human language reveals statistical relationships across domains that were invisible to the humans who generated the language. Call it the unreasonable effectiveness of compressed human expression. The map, when viewed at sufficient scale, reveals candidate coastlines that none of the original surveyors drew intentionally, though, following Wigner's own caution, some of these coastlines may be artifacts of how humans happen to describe things rather than features of the territory.
A coding agent that has ingested thousands of codebases does not merely know more patterns than a single architect. Pan et al. (2025) demonstrate empirically that code LLMs can recognise design patterns across codebases, though with systematic biases in generation and comprehension, a finding that supports both the binding-substrate capability and its limits. The agent holds thousands of architectural patterns simultaneously in the same representational space, which means it can detect architectural anti-patterns, design rhymes, and structural trade-offs that a human architect would need decades of cross-project experience to develop partial intuitions about. It can zoom out in a way that no single human can, stitching together design principles from distributed systems, compiler theory, database internals, and UI architecture into a coherent structural recommendation, not because it understands any of these fields in the way their practitioners do, but because it perceives geometric relationships between them that the practitioners, each immersed in their own regional chart, cannot see. The agent's reach here is bounded by its training distribution: proprietary codebases written in uncommon patterns or underrepresented languages will have thinner geometric structure, and the binding substrate offers less advantage in those regions.
The economics of complementary knowledge
This is not intelligence beyond the map. It is intelligence of the map that the mapmakers, each holding only a fragment, could never achieve. The bird's-eye view is genuinely new. No prior system has held millions of documents in a single geometric space where statistical similarity is encoded as distance and traversable in a single computational pass. Libraries and encyclopaedias provide indexed access; polymathic individuals provide cross-domain intuition. Neither provides simultaneous, geometrically structured access to the full surface of recorded human knowledge.
Economic complexity research (Hidalgo, 2015) helps explain why this simultaneous access generates such disproportionate value. Knowledge, at every scale from the individual to the economy, is non-fungible: its value comes not from volume but from complementarity. Adding a hundred people who know the same thing increases capacity linearly. Adding a hundred people with different, complementary knowledge creates combinatorial possibilities that grow exponentially, what Hidalgo calls the "infinite alphabet" of productive knowledge. Human organisations have always struggled with this combinatorial potential because complementary knowledge is distributed across specialists who communicate through the narrow bandwidth of meetings, documents, and handoffs. The binding substrate changes the economics fundamentally. It holds the full alphabet (the non-fungible, complementary knowledge of every domain in the corpus) in a single geometric space where statistical co-relevance between domains is structurally visible as distances and directions. When the agent identifies that a technique from supply-chain optimisation might apply to neural architecture search, it does so because the statistical signatures of how humans describe both domains share geometric structure, a proxy for functional complementarity, though not a guarantee of it. The agent does not need a cross-functional meeting to surface these candidates. Whether the candidate complementarity is genuine requires verification against the territory, but the generation of candidates at this scale and speed is itself transformative.
Wigner's lecture on mathematics carried a further caution that transfers directly. He noted that the effectiveness of mathematics might be partly a selection effect: we notice the cases where formal structures map onto reality and overlook where they do not. The same selection effect applies to the binding substrate. When the agent surfaces a cross-domain connection that proves valuable, we celebrate the capability. When it surfaces a connection that is merely linguistic, metaphorical convergence mistaken for structural convergence, we dismiss it as noise. The true hit rate is an empirical question that the essay cannot settle. What can be said is that the agent, operating entirely within the map, cannot distinguish genuine structural isomorphisms from linguistic artifacts. That distinction requires someone who can check the map against the field.
3. The Boundary
For current architectures trained primarily on text, the boundary of the agent's conceptual reach is set by the interaction of the three constraints identified in §1: what its signal source contains (provenance), whether its abstraction layer can reorganise around new signal (it cannot, intrinsically), and how tightly its representations are coupled to action in the world (domain-dependent: tight in formal territories like code and mathematics, weak in unformalised ones). Certain kinds of knowledge require direct encounter with the territory, not refinement of existing maps. (The boundary is not permanent or metaphysical; §8 discusses how multimodal and embodied architectures progressively shift it. But for the dominant class of text-trained systems, the boundary is real and has engineering consequences that the rest of this section develops.)
A novel conceptual primitive, as the term is used here, means a concept for which no existing term or combination of terms in the training vocabulary captures the referent, one generated by encounter with phenomena that cannot be described as a composition of existing symbols. The distinction from recombination is operational: if the concept can be expressed as a function of existing vocabulary, it is a recombination; if expressing it requires minting a new symbol because the existing vocabulary lacks the distinctions needed, it is a primitive. An important honesty: the boundary between sophisticated recombination and genuine primitive is a continuum, not a bright line. Ward Cunningham's coining of "technical debt" sits near the borderline: the metaphorical mapping from financial debt to design shortcuts is the kind of cross-domain structural isomorphism that the binding substrate handles well, and an agent might plausibly propose the term. What the agent would not do is notice the phenomenon in the first place, the specific experiential pattern of accruing shortcuts under deadline pressure and paying compounding costs later, without having encountered it through practice. At the far end of the continuum, the distinction is clearer. When Barbara McClintock identified transposable genetic elements, she was observing phenomena (chromosomal patterns in maize that contradicted the static-genome model) for which no existing vocabulary provided even the analogical raw material. The concept required direct encounter with the territory and was resisted by the field for decades precisely because the inherited vocabulary had no place for it.
What lies beyond the boundary, then, is not a single category but a class of knowledge that shares a common feature: it cannot be generated by operating within the existing vocabulary, however sophisticated the recombination, because the vocabulary lacks the distinctions needed to even describe the relevant phenomenon. The model knows what humans have written about physics, not what physics does in cases the writing does not cover. The model can identify inconsistencies between frameworks (and, as §2 argues, does so better than any individual human), but cannot determine that the reason for the inconsistency is a gap in the joint vocabulary, a concept missing because no one has yet encountered the territory it would describe. The binding substrate's hypothesis-generating power operates up to the edge of the vocabulary; beyond that edge, the generation of new hypotheses requires new primitives, which require territory encounter.
This boundary claim is structural rather than empirical: it follows from the three-part constraint (§1) rather than from observed failure cases. The remainder of this section first examines the strongest challenges to the claim (self-evolving agents, the formalisation continuum, the dehydration of embodied knowledge) and then turns to three symptom classes (hallucination, reward hacking, and the Fluency Trap) that provide converging evidence the boundary has practical implications.
Self-evolving agents and the recombination ceiling
The recombination ceiling marks the difference. Interpolation and extrapolation within a concept space are powerful, and as the previous section argued, the agent performs them at a scale and resolution no human can match. The higher-order structures acknowledged in §1 (latent representations that reorganise and extend the inherited vocabulary) are recombinations in this sense, however novel their form. The boundary concerns primitives: extending the space itself, adding a genuinely new primitive, is structurally different from recombining existing ones.
The strongest challenge to this claim comes from recent work on self-evolving AI systems. Sakana AI's AI Scientist (Yamada et al., 2025) and ShinkaEvolve (Lange, Imajuku, and Cetin, 2025) demonstrate agents that maintain evolving populations of programs, mutate and recombine code through evolutionary search, and autonomously generate scientific papers that survive peer review. Karpathy's autoresearch (2026) takes the same principle in a more focused direction: an AI agent is given a small but real LLM training setup and left to experiment autonomously overnight, modifying training code, running five-minute experiments (time capped intentionally to act as a constraint), evaluating results, and deciding whether to keep or discard changes in a continuous loop. These systems genuinely intervene and learn: they design experiments, run code, observe results, and iterate, closing a real feedback loop with a territory. This is not spectator epistemology. It is haptic engagement with a computational environment, and the results are impressive. In the terms of §1's three-part constraint, these systems have tight action coupling to their formal territory: the compiler, the test suite, the training loss. Their impressive performance is predicted by the framework: all three constraints are favourable in formal domains (rich signal source from the corpus, latent representations that align well with formal structure, and tight action coupling through execution environments).
The question is whether these systems extend the conceptual vocabulary or explore it more thoroughly. To answer it without begging the question, we need an independent test. Consider: the AI Scientist generates papers by recombining techniques from existing ML literature and running experiments in code execution environments. The vocabulary of those experiments (loss functions, architectures, benchmark metrics) was defined by human computer scientists. The system discovers novel combinations (a new architecture variant, a new training schedule) and validates them against formal criteria (benchmark performance). What it has not demonstrated is minting a concept for which no existing term or combination of terms suffices, the kind of primitive that requires encounter with phenomena outside the system's formal vocabulary. The "problems" the system invents for itself are novel configurations within an existing conceptual space, not encounters with physical or social reality that demand new symbols.
This is not a definitional dodge. It is an observable distinction: we can ask whether the system's outputs are expressible in the vocabulary it inherited or whether they require new primitives. So far, the outputs are expressible: impressively novel combinations, but combinations nonetheless. Most tellingly, these systems require what their creators call a human "shepherd" who maintains the overarching value function and grounds the search in domains that actually matter, deciding which regions of the map correspond to territory worth investigating. Self-evolving agents dramatically raise the recombination ceiling. They close loops within formal territories with increasing sophistication. Whether an embodied agent with sufficient sensory access could use evolutionary search to actually generate new vocabulary, minting primitives from direct encounter with physical or social territory, is an open empirical question that §8 addresses. The claim here is narrower: current self-evolving systems, impressive as they are, operate within inherited formal vocabularies.
Haptic realism and dehydrated knowledge
Chirimuuta (2024) offers a direct philosophical framing of what lives on the other side of this boundary. Her distinction between haptic realism and spectator epistemology maps onto the asymmetry between mapmaker and cartographer. The binding substrate, even when it closes loops with formal territories as Sakana's agents do, remains spectator to what we might call the unformalised territory: domains of reality that have not been fully encoded in the symbolic systems (natural language, mathematics, code) present in the training corpus. Physics is substantially formalised; social dynamics less so; novel phenomena, the not-yet-named, are entirely unformalised. The distinction is not binary but a continuum of formalisation, and the agent's reach extends as far as the formalisation does.
An important nuance: "direct encounter" with the territory is itself a continuum, not a binary. A robot's camera feed is more direct than a journal article describing an experiment; a journal article is more direct than a secondhand summary. The relevant distinction is not sensory immediacy per se but whether the signal source includes information not already encoded in the training vocabulary. Give the agent a camera, a robot arm, a database connection, a set of sensors, and it gains access to signal sources beyond the training corpus, progressively expanding its signal base. Whether that richer signal translates into genuine vocabulary extension depends on whether the abstraction layer can reorganise around it (the distinction developed in §1 and below via Bennett (2024) framework). Empirical verification (running the code, querying the database, measuring the physical output) is increasingly automatable as sensor integration improves. What remains beyond automation is not the check itself but the judgment about what to check and whether the check is meaningful: deciding which measurement matters given what is at stake, which is a function of the mapmaker's situatedness and consequence-bearing rather than of sensory access.
The concepts in the agent's vocabulary were originally produced through haptic engagement (through experiment, craft, embodied trial) but the agent inherits only the linguistic residue, not the generative process. The vocabulary is dehydrated: compressed from rich, embodied, interventionist encounters into tokens. The agent can rehydrate those tokens into fluent output, but it cannot regenerate the encounters that gave each token its original referent.
The dehydration metaphor is pragmatic due to an empirical regularity: knowledge decays without embodied exercise. Hidalgo (2015) documents this at every scale. An individual forgets a language they no longer speak. An organisation loses a manufacturing capability even when the blueprints and machines remain. A region's industrial expertise atrophies when the practitioners retire. The canonical example is NASA's loss of the tacit knowledge required to build the Saturn V rocket: the blueprints survived, but the workforce that understood how to interpret and execute them dispersed, and rebuilding the capability decades later required relearning from scratch rather than simply re-reading the documentation. The decay rate is steep: inactive knowledge can halve within a year. The training corpus is a snapshot of knowledge that was alive at the moment of expression. Some of that knowledge has since decayed in the humans and institutions that produced it. The agent preserves the tokens indefinitely (it does not forget) but the tokens are fossils of a living process. The agent's perfect retention masks the fact that what it retains is not knowledge in the embodied, exercised sense but the linguistic trace of knowledge that once was. This is a further reason why the boundary is architectural: the agent's vocabulary is not merely inherited but fossilised, preserved in form but disconnected from the living practice that gave it meaning. Bennett's (2024) framework gives this observation formal teeth. Minting a new conceptual primitive requires more than access to new data; it requires an adaptive abstraction layer that can shift its inductive biases in response to environmental interaction. Biological mapmakers have such layers (cells, tissues, neural circuits that reorganise through use). The foundation model's abstraction layer does not intrinsically reorganise through environmental interaction (in-context learning adapts computation within a forward pass, but the underlying computational structure remains fixed). Expanding the signal source (multimodal input, sensor feeds) is necessary but not sufficient for vocabulary extension. The abstraction layer must also be able to reorganise around the new signal, forming new representational invariants rather than merely compressing the new signal through existing ones. Dupoux, LeCun, and Malik (2026) confirm this from a developmental perspective: biological learners inherit inductive biases (the evolutionary component) but also develop new representational structures through environmental interaction (the developmental component). Current AI has the evolutionary part (architecture design, pretraining recipes authored by human engineers) but not the developmental part. The abstraction layer is initialised but not grown.
Three symptom classes
The boundary shows up practically, as can be illustrated in at least three symptom classes.
Hallucination is the most discussed: the model generates plausible outputs that violate factual reality because it is operating in the map (statistical regularities of text), not the territory (the world those texts describe).
Reward hacking is the second: the model optimises the proxy (the map) because it has no access to the referent (the territory). The first essay's analysis of dimensional reduction, collapsing plural values into a scalar, is a special case of the simulacra boundary: the scalar is a map of the map, doubly removed from the territory.
The third symptom class is the most revealing. Best seen as a Fluency Trap: perfect execution with zero semantic judgment. A model asked to refactor a building automation codebase restructures the control logic with impressive fluency, improving modularity, readability, and test coverage, while faithfully preserving a subroutine that compensates for a sensor fault repaired six months ago. The compensation is syntactically correct and internally documented ("adjusts for Unit 7 drift"). Nothing in the codebase records that the physical repair happened, because it happened in the building, not in the repository. The model refactored the map with precision; recognising that a workaround has outlived the fault it addresses requires knowledge of the physical system the code controls, territory that no amount of textual reasoning can reach.
This example resolves an apparent contradiction. §2 claims the agent has unprecedented simultaneous access to the full surface of human knowledge. Here, the agent misses a semantic problem that any practitioner would catch instantly. These are not contradictory because they operate at different levels. The binding substrate operates at the structural level: geometric relationships between domains, architectural patterns, cross-field isomorphisms. The fluency trap operates at the semantic level: whether a particular subroutine still corresponds to a condition in the physical world. The agent can see the entire coastline of the map while being unable to tell whether a particular harbour still exists, because the map's geometry encodes relationships between concepts but not whether the territory has changed since the map was drawn. This is the sharpest illustration that the boundary is not about capability gaps (which scale away) but about vocabulary provenance (which does not).
The common observation that AI capability is a "jagged frontier" (brilliant at some tasks, baffling at others) is worth interpreting through this lens. Much of the jaggedness is an organisational artifact: it arises from forcing models into single-turn, zero-shot interactions without structural support. When agents are given time to reason, tools to use, and the ability to retry, the capability surface smooths out universally within the map. The remaining jaggedness, the irreducible unevenness, marks the actual vocabulary boundary. It is a topographical map of where the inherited concepts run out.
Why this is not the Chinese Room
Searle's Chinese Room argument claims that syntactic manipulation cannot produce understanding, a consciousness claim. This essay makes a different, more limited, and more practically useful claim, closer to Chirimuuta's (2024) than to Searle's: you cannot safely prune away the physical, embodied, historical process that generated a system's concepts and expect the abstracted residue to retain semantic reach beyond those concepts. The boundary identified here is not about whether the agent "understands" in any philosophically interesting sense. It is agnostic about consciousness, qualia, and subjective experience. It is about provenance: the agent's vocabulary was produced by embodied beings through haptic engagement with the world, and the agent inherits the vocabulary without the engagement. That is an engineering-relevant observation, not a metaphysical one. (A note on the map/territory analogy itself: it is useful for framing provenance but imperfect in its details. Unlike a literal map, the agent's representation is a dynamic computational space, not a static document. Unlike a literal territory, "the world" encompasses everything from physical systems to social dynamics to formal mathematics. The analogy carries the essay's argument about provenance and boundary; it should not be pressed to carry claims about the agent's internal computational mechanics, which are better understood through the latent-space framework of §1.)
4. The Execution Substrate
Because the agent is fluent across the entire inherited vocabulary, it becomes a universal execution substrate, and this is where the operational consequences of the provenance claim play out.
Guo et al. (2024), surveying multi-agent LLM systems across laboratories, document convergence on a shared architectural pattern: decompose the problem, parallelise execution, verify outputs, iterate toward completion. Systems as different as MetaGPT (Hong et al., 2023), AutoGen (Wu et al., 2023), and CrewAI (Moura, 2024) independently arrived at similar role-differentiated, plan-execute-verify loops. This convergence is partly predicted by the provenance argument. If all agents inherit similar vocabulary from overlapping corpora, compressed through similar architectures, then the coordination patterns that work well for one system will tend to work well for others, because the map is shared. Jiang et al. (2025) document this convergence empirically as the Artificial Hivemind effect: models trained on similar data produce strikingly similar outputs and, by extension, similar coordination dynamics. The convergence is not total; differences in fine-tuning objectives, training data composition, and deployment context produce meaningful variation, but the family resemblance across laboratories is strong enough to constitute a pattern.
Within this substrate, capability is not a fixed property. It is a state-dependent execution mode. Wei et al. (2022) document that capabilities emerge discontinuously with scale: abilities absent in smaller models appear suddenly in larger ones. The same principle applies at runtime: the gap between a model's low-compute and high-compute states demonstrates that resource allocation dynamically alters epistemic calibration. Tasks that a system handles correctly with extended reasoning chains may produce confident hallucinations when the same system operates under compute constraints. Intelligence, for the execution substrate, is a function of runtime depth, not just pre-trained weights, though the baseline calibration depends on the training regime, and scaling compute alone does not fix miscalibration baked in during training. In all cases, what the substrate processes remains the inherited vocabulary: deeper computation reveals more of the map's latent structure without extending the map itself.
The substrate also discovers its own toolkit at runtime, extending its operational reach within the inherited vocabulary on the fly. Static, upfront provisioning of tools (loading all API definitions into context before execution) creates scaling bottlenecks. The architectural shift, already visible in deployed coding agents (Claude Code, Codex, and similar systems), is toward progressive, runtime tool discovery: when a system encounters an unsupported file type or missing dependency, the capable agentic behaviour is not to skip it but to dynamically fetch and install the required capability mid-flight. System design moves from monolithic toolkits to composable micro-capabilities discovered just in time. Agents also generate bespoke tooling on the fly, writing small scripts and utilities to bridge gaps no pre-built tool covers. This is genuine creation, but it is creation within the inherited vocabulary: the agent recombines known patterns, APIs, and programming constructs to produce new artifacts. It extends its operational reach, not its conceptual vocabulary.
For scaling multi-agent execution to complex tasks, hierarchy and isolation outperform flat coordination. Mieczkowski et al. (2026) formalise this insight by treating LLM teams as distributed systems, showing that the same fundamental challenges studied in distributed computing (consensus, fault tolerance, coordination overhead) arise in multi-agent LLM deployments. In multi-agent systems, flat coordination, where agents share files or states, leads to risk aversion and collisions. Progress scales through strict hierarchy: planners decompose tasks, workers execute in isolated sandboxes, and judges evaluate without contaminated context. Both human organisations and agent swarms share fundamental constraints: finite context, imperfect memory, variable per-step reliability. Overcoming these limitations relies on organisational design (roles, handoffs, verification steps) rather than isolated genius. Intelligence, at scale, is management. That this maxim applies equally to human teams and agent swarms is itself a consequence of the provenance claim: both navigate the same inherited vocabulary under analogous information-theoretic constraints.
All of this describes what the cartographer does brilliantly within the map. And it commoditises. Because the vocabulary is shared and the coordination patterns converge, the execution layer becomes infrastructure. The competitive advantage, the irreducible source of differentiation, must come from somewhere else.
5. Vocabulary-Aware System Design
The companion essay argued for evaluator stacks, governed boundaries, and layered objectives. The provenance claim adds a specific constraint: the seam between human and agent is not just an approval gate. It is a concept-transfer interface, and it must be typed to the vocabulary boundary.
The practice of defining these interfaces is architectural seam design: strict, typed boundaries between human judgment and agent execution. Instead of treating workflows as unstructured monoliths, operators decouple them into distinct sub-routines, specify which are agent-executable, and design the structural artifacts, the seams, that pass between them. These seams function as verifiable endpoints where a human can run a rapid check before the next phase begins. The agent pauses not merely when it is uncertain in a probabilistic sense, but when it reaches the edge of its conceptual vocabulary, when the task shifts from map-level execution to territory-level judgment.
Honest system design, in this framing, means being explicit about what the inherited vocabulary covers and where its boundaries lie. This essay interprets hallucination, reward hacking, and the Fluency Trap not as isolated bugs to be patched but as symptomatic of the provenance boundary, cases where the model's statistical map diverges from the territory it is being applied to. This is a theoretical reframing, not an empirical finding, but it has a practical payoff: it suggests that the evaluator stack should include checks for vocabulary adequacy: whether the agent's inherited vocabulary is rich enough for the domain it is being deployed into. Such a check would ask: does the training corpus contain sufficient coverage of this domain's concepts, edge cases, and failure modes? If not, the system should route to a human at a lower threshold, regardless of the agent's confidence. The precise procedure for assessing vocabulary adequacy remains an open engineering problem, but the framing identifies what needs to be checked.
Designing for the boundary requires two further practices.
The first is topographical failure modelling. Generic scepticism of AI output is as inefficient as a naked catch-all exception handler. Effective operators maintain differentiated mental models of how current models fail: knowing, for instance, that an agent might handle standard data transformations perfectly but silently hallucinate on implicit nulls or mixed data types. Maintaining this typed failure model allows for targeted verification rather than exhaustive review. It is, in effect, a topographical map of where the inherited vocabulary runs thin.
The second is latency-versus-completeness arbitration. In autonomous, long-running processes, organisations must trade execution latency for output completeness. A system optimised for exhaustive coverage provides depth but sacrifices the tight feedback loops required for human-in-the-loop iteration. Models optimised for concise synthesis drop edge cases entirely. The architectural choice is a seam-design decision: where in the workflow does the human need to intervene, and how much execution should the agent complete before reaching that seam?
Multi-agent harnesses can operate autonomously in any domain where output is objectively verifiable: machine-checkable tasks (unit tests passing, type checks succeeding) and expert-checkable tasks (where seasoned professionals can reach consensus on correctness). Where verifiability breaks down, where checking the output requires territory-level judgment that the map cannot supply, the seam must route to a human. The evaluator stack, from the first essay, is the mechanism. The vocabulary boundary, from this essay, tells you where to place it.
6. The Economics of the Mapmaker
With execution commoditised and the binding substrate operating at unprecedented scale, the mapmaker's role transforms. The transformation is not a diminishment. It is a concentration.
Much of what constitutes "work" in a modern organisation (for example, in software development: product requirement documents, sprint planning, status synchronisations, design handoffs) exists not to create value directly but to synchronise state across multiple human nodes, each suffering from limited bandwidth and constrained context windows. Call this the coordination tax: the fraction of organisational effort spent translating, aligning, and reconciling information between people rather than producing the end artifact. Not all coordination is tax; some alignment is genuinely necessary for quality and coherence. But the portion that exists solely because humans cannot share context at high bandwidth is overhead that the agent substrate can absorb. The agent substrate, capable of near-zero-latency context sharing and instantaneous state synchronisation, renders this human-to-human translation layer largely obsolete.
As the coordination layer dissolves, the distance between human intent and the final product collapses. The trajectory, already visible in early adopter organisations using agent-assisted development, is from drafting approximations (for example, specifications, wireframes, briefs) to iterating directly on the terminal artifact: the codebase, the live product, the actual document. This is artifact proximity: the progressive elimination of translation steps between what the human means and what gets built. How far this trajectory extends, and how quickly, remains to be seen. But the direction is clear.
Organisations have traditionally conflated the judgment layer (strategic vision, problem definition) with the execution layer (coordination, implementation). Agent harnesses allow these to be cleanly unbundled. By migrating the execution substrate from human teams to agent loops, human effort concentrates not on "judgment" in some broad, unexamined sense but on three specific functions the agent structurally cannot perform.
The first is vocabulary extension: encountering the territory directly and minting new conceptual primitives. Not recombining existing terms (the agent does that better than any human) but creating a new symbol because you have encountered something no existing symbol captures. A clinician who notices a pattern in patients that no textbook describes and names it. A software architect who recognises that a recurring class of failures needs a new abstraction (like "technical debt" before Ward Cunningham coined it). The agent can propose candidate terms by recombination, and its binding substrate makes it extraordinarily good at this. But validating that a candidate term corresponds to something real in the territory requires someone who can visit the field. Territory encounter is the mechanism observed in every historical case of genuine conceptual primitive minting the essay can identify. Whether sufficiently rich inference from existing data could, under some conditions, achieve the same result is an open empirical question. The essay's claim is not that alternative mechanisms are impossible in principle, but that none has been demonstrated, and that the architectural constraints identified in §1 (static abstraction layer, weak action coupling) explain why.
The second is consequence-bearing commitment: making decisions whose outcomes are irreversible and personally costly, where "costly" means the decider's future options, reputation, resources, or wellbeing are materially altered by the outcome. The agent can recommend, rank options, and model tradeoffs. It does not live with the result. The human who signs the contract, ships the product, publishes the paper, or abandons a line of research activates a mode of scrutiny that costless processing cannot replicate. Knowing that you will bear the consequences of a decision changes how you evaluate the options. It is epistemically productive, not merely a legal formality: personal stakes change the decision-maker's attention allocation, risk weighting, and willingness to seek disconfirming evidence. This does not mean every decision benefits from consequence-bearing: routine choices where the agent's recommendation is verifiable add friction without epistemic gain if routed to a human. The claim is narrower: that for decisions involving irreversibility, ambiguity, and significant stakes, the decider's skin in the game measurably changes the quality of scrutiny applied.
The third is priority arbitration under stakes: deciding what matters, from a position where the decision has consequences for the decider. The agent can identify that seventeen things are relevant and rank them by twelve different criteria. The human decides which three to pursue given finite time, finite budget, political context, and personal exposure. This function is closely related to consequence-bearing: you arbitrate priorities differently when you bear the results, though the arbitration skill itself (what to cut, what to pursue, what to defer) is not reducible to consequence-bearing alone. The agent optimises within a value function. The human authors the value function, and revises it when reality pushes back. The distinction is sharpened by recent work showing that AI can learn "scientific taste," predicting community-level evaluative preferences with impressive accuracy (Tong et al., 2026). But learned taste is the statistical regularity of what the community has valued, compressed from citation signals. Priority arbitration under stakes is a different operation: deciding what to pursue from a position where the decider's future is materially altered by getting it wrong. The agent can predict consensus; it cannot author a value function from a stakeholding position.
These three functions are derived from this essay's theoretical framework (provenance, binding, finitude) and represent the most salient partition given that framework. Other framings, rooted in creativity, aesthetic judgment, or moral authority, might identify overlapping or additional functions. The claim is not exhaustiveness but structural importance: these three follow from the architectural analysis and capture the functions most directly implicated by the provenance boundary.
There is a deeper point here that Chirimuuta's (2024) framework shows. The mapmaker's bounded attention (finite working memory, domain-specific expertise, embodied situatedness) is not a bottleneck to be optimised around. It is the source of evaluative judgment. Her principle that finitude is a feature, not a bug, applies directly: meaning and salience are generated by finite systems with stakes. The agent's attention is cheap precisely because nothing depends on where it falls. It processes tokens at negligible marginal cost, with no irreversible consequences for itself regardless of the outcome. The mapmaker's attention is expensive because it is consequential, situated in a world where decisions bind, resources deplete, and errors compound. That asymmetry is not a deficiency in the human-agent system. It is the system's epistemic engine. The agent provides breadth and binding; the mapmaker provides the evaluative gravity that only a finite, situated, stakeholding system can supply.
7. The Moving Frontier
The boundary is not static. It is an expanding membrane. The volume inside represents fully automated, agent-executable tasks. The surface is the active frontier of human-agent collaboration. The space outside is territory-level judgment. Because the internal volume expands with every model generation (larger context windows, better retrieval, new modalities), static workflow definitions are continuously deprecated. The mapmaker who calibrated their assumptions against a model's failure rate six months ago is operating on stale data. Staying relevant requires migrating outward with the membrane, continuously probing the boundary through intentional experimentation: feeding agents novel inputs specifically to discover where execution fails, and updating one's internal model accordingly. The frontier does not necessarily move smoothly inward: as agents absorb more execution, the verification burden may increase in complexity even as it decreases in volume, and the surface area of the membrane (the number of seams requiring human judgment) may grow even as its interior expands.
Movement through knowledge space, however, is constrained by adjacency, what Hidalgo (2015) calls relatedness in the product space. Whether an individual is learning a new skill or an economy is entering a new industry, the probability of success depends on distance from current capabilities. Aircraft manufacturers can transition to motorcycles because the technical capabilities are adjacent; they cannot leap to pharmaceuticals. The same constraint applies to the expanding membrane, with an important asymmetry. The agent's binding substrate is better than any individual human at detecting cross-framework inconsistencies and connections between adjacent capability regions; this is the bird's-eye view at work. Given the full map, the agent can identify that two frameworks conflict, that a technique from one domain resolves a bottleneck in another, that an architecture pattern has structural parallels across industries. No single human, constrained by working memory and domain-specific expertise, can match this breadth of detection. The agent's superiority at within-map pattern recognition is real and should not be understated. A caveat: the agent's bird's-eye view is a weighted view, skewed by the training distribution. Well-represented domains have rich geometric structure; underrepresented ones have thin structure. The agent's detection of cross-framework connections will be biased toward the well-represented regions, potentially missing genuine connections that involve poorly documented domains.
What the agent cannot do is detect that the vocabulary itself is inadequate, that the reason two frameworks conflict is that neither possesses a concept for something that exists in the territory but has never been named. This distinction requires care. If two frameworks conflict and a third framework in the corpus resolves the conflict, the agent can find it; this is standard within-map pattern matching. The harder case is when no framework in the corpus possesses the relevant concept. The agent can say "these two maps disagree about this coastline." It cannot say "the reason they disagree is that there is a river here that neither cartographer noticed, because neither visited this region." An important qualification: the agent can potentially detect symptoms of vocabulary inadequacy, persistent anomalies, contradictions that resist resolution within the existing framework, systematic prediction failures in a region of the map. What it cannot do is diagnose the cause as a missing primitive or generate the primitive itself, because generating a new primitive requires encounter with whatever the existing vocabulary fails to capture. The claim is narrower than "the agent cannot notice something is wrong." It is that the agent cannot determine what is wrong when the answer is "the conceptual apparatus needs a term for something it has never encountered." A further honesty is warranted. The latent reasoning evidence from §1 (Rofin et al., Hao et al.) suggests the agent's internal representations encode structure beyond what the explicit vocabulary contains. The agent may, in effect, detect a gap-shaped anomaly in its latent space, a region where prediction fails systematically or where geometric structure is thin, without being able to articulate what is missing. This is closer to what humans experience pre-verbally when they sense that something does not fit. For the vast majority of practical situations, this latent detection combined with a human's ability to follow up on flagged anomalies covers the ground. The genuinely irreducible case is narrow: situations where the phenomenon is so far from the training distribution that even the latent geometry is silent, and only direct territory encounter can reveal it. These cases are rare, but they are the cases that determine whether the system is honest about its limits. The boundary is not wide. Its narrowness is what makes it important: paradigm-level vocabulary extension, for current architectures, remains the one operation that requires encounter with the territory and cannot be approximated by any amount of within-map processing, however deep.
In an agent-rich environment, human attention becomes the sole blocking thread; agents are the asynchronous workers. As the ratio of agents to humans scales, reviewing all outputs at equal depth creates a bottleneck. Operators must implement hierarchical load balancing for their own attention: automating verification for routine paths via test suites, spot-checking medium-risk outputs, reserving deep human cycles for high-variance architectural decisions. Attention is the scarcest resource. Its allocation is the highest-leverage decision the mapmaker makes.
There is a tension here that this essay can identify but not resolve. The three irreducible functions are not static endowments; they are skills that require cultivation. And the traditional cultivation path, learning vocabulary extension, consequence-bearing, and priority arbitration through years of hands-on execution, is precisely what agent-mediated work disrupts. The Saturn V example from §3 applies at the individual level: if the mapmaker delegates all execution to the cartographer, the embodied practice that sustains territory contact atrophies. Maintaining the capacity to perform the irreducible functions while the execution substrate absorbs the activities that historically developed that capacity is a problem of professional formation, not just system design. A companion essay will take up this question directly: what the mapmaker's own development path looks like when the traditional apprenticeship through execution is no longer the default.
A further elaboration supports the analysis. The three functions are not equally contingent on current architecture. Vocabulary extension is architecturally contingent: if the abstraction layer becomes adaptive and the action coupling tightens (Bennett's horizon, discussed in §8), embodied agents may genuinely mint new primitives. Consequence-bearing commitment and priority arbitration under stakes, by contrast, are structurally irreducible in a stronger sense: they require not just territory access but stakeholding, the condition of being a finite system whose future is materially altered by its own decisions. No architecture, however adaptive, provides stakeholding; it provides processing. The distinction between architecturally contingent and structurally irreducible is important because it tells practitioners where to invest: vocabulary extension skills may be overtaken by engineering progress, while consequence-bearing and priority arbitration are likely to remain human functions regardless of architectural advances.
The deepest form of the human-agent collaboration loop now comes into view. The agent, operating as a binding substrate across the full inherited vocabulary, surfaces candidate insights: structural isomorphisms across domains, architectural patterns, latent connections. The human mapmaker's role is to test those candidates against the territory: to determine whether a detected similarity reflects something real in the world or is an artifact of the map's compression. The insights that survive become new conceptual vocabulary. The agent's next generation inherits a richer map. The mapmaker draws the symbols; the cartographer reveals that two distant coastlines are actually the same continent; the mapmaker sails out to check.
8. The Cartographer's Integrity
This essay has traced a single arc. The agent's conceptual vocabulary is inherited, not self-authored: dehydrated from embodied encounters into tokens, fossilised in form while the living practice behind it decays. Within that vocabulary, the agent operates as an unprecedented binding substrate, holding the entire map in geometric space and detecting structural relationships invisible to any regional specialist. The intelligence framework from §1 explains why the boundary holds where it does. The agent compresses brilliantly within its signal source (Ma), learns rapidly across the inherited vocabulary (Chollet), and extends its effective reach across the full surface of the map (Levin). But its abstraction layer (Bennett) does not intrinsically reorganise through environmental interaction (in-context learning simulates adaptive bias; it does not restructure the computational substrate), and it remains weakly coupled to action in the world. The three constraints from §1 (signal source, abstraction layer, action coupling) converge to explain why, at the edge of the vocabulary, three irreducible functions remain with the mapmaker, each rooted in what Chirimuuta (2024) identifies as the epistemic advantage of finitude: vocabulary extension, consequence-bearing commitment, and priority arbitration under stakes. The practical question is not whether the boundary exists but how fast it moves, and what honest engineering looks like in the meantime.
The boundary is not permanent or metaphysical. It is a consequence of current architecture, and the engineering paths toward narrowing it are already visible. Multimodal models begin to address it by giving the agent access to non-linguistic regularities. Embodied agents go further by allowing direct encounter with the territory. Self-evolving agents close loops with increasingly complex formal domains. These connect to the first essay's framework: the agent's compression function operates over whatever signals it receives, and expanding the signal space expands the reachable concept space. Bennett's framework suggests the deeper shift: if the abstraction layer itself becomes adaptive rather than static, the boundary moves not incrementally but architecturally. Grounding, in the engineering sense, is vocabulary extension. A caveat: embodied access is itself mediated. Sensors have their own biases and sampling constraints, and whether embodiment achieves genuine vocabulary extension or merely expands the signal source within the existing abstraction layer is an open question. But the direction is clear. Dupoux, LeCun, and Malik (2026) offer the most concrete architectural roadmap: integrating observational learning with action-based learning under autonomous meta-control, and bootstrapping the combined system through an evolutionary-developmental framework that jointly optimises architecture and learning strategy. If such a system were realised, it would address all three constraints simultaneously: richer signal sources (multimodal, embodied), tighter action coupling (learning through intervention), and an adaptive abstraction layer (developmental reorganisation through environmental interaction). The more territory the agent can access directly, and the more autonomously it can reorganise its own learning around that access, the narrower the residual human role becomes.
What matters, then, is the integrity of the boundary specification: how honestly the system declares where agent execution ends and human judgment begins. A system that claims full autonomy where vocabulary extension is required will fail silently: it will produce outputs that are fluent, internally consistent, and wrong in ways that only territory-level contact can reveal. A system that routes every decision to a human "for safety" wastes the binding substrate and reimports the coordination tax that agents exist to dissolve. The cartographer's integrity lies in neither overstating nor understating the map's coverage. The most consequential engineering decision is not how capable the agent is. It is how honestly the system specifies where the map ends, and ensures that a mapmaker is present at those seams.
Relationship to "The Agent, Not the Task"
This essay is a companion. Where the first argues for how to build agents (compression, adaptation, layered objectives, governed evaluators) this essay asks what kind of thing the agent is and what follows from the provenance of its representations. The first essay's framework remains intact but gains three additional constraints: the compression function operates within a vocabulary it did not author; within that vocabulary it perceives structural relationships that its authors could not, functioning as a binding substrate of unprecedented power; and the human role, once everything automatable has been automated, reduces to three specific functions (vocabulary extension, consequence-bearing commitment, and priority arbitration under stakes) rather than the broad category of "judgment," with vocabulary extension being architecturally contingent and the latter two structurally irreducible. The engineering agenda is not diminished by these observations. It is better focused. Knowing both the reach and the boundary of the inherited vocabulary is what allows you to build systems that exploit the binding substrate fully while remaining honest about where the map ends and the territory begins.
References
Bennett, M.T. (2024) 'Are biological systems more intelligent than artificial intelligence?', arXiv preprint, arXiv:2405.02325.
Chirimuuta, M. (2024) The Brain Abstracted: Simplification in the History and Philosophy of Neuroscience. Cambridge, MA: MIT Press. doi:10.7551/mitpress/13804.001.0001.
Chollet, F. (2019) 'On the measure of intelligence', arXiv preprint, arXiv:1911.01547.
Cowan, N. (2001) 'The magical number 4 in short-term memory: A reconsideration of mental storage capacity', Behavioral and Brain Sciences, 24(1), pp. 87–114. doi:10.1017/S0140525X01003922.
Dupoux, E., LeCun, Y. and Malik, J. (2026) 'Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive science', arXiv preprint, arXiv:2603.15381.
Elhage, N., Hume, T., Olsson, C., Schiefer, N., Henighan, T., Kravec, S., Hatfield-Dodds, Z., Lasenby, R., Drain, D., Chen, C., Grosse, R., McCandlish, S., Kaplan, J., Amodei, D., Wattenberg, M. and Olah, C. (2022) 'Toy models of superposition', Transformer Circuits Thread, arXiv:2209.10652.
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N.V., Wiest, O. and Zhang, X. (2024) 'Large language model based multi-agents: A survey of progress and challenges', Proceedings of the 33rd International Joint Conference on Artificial Intelligence (IJCAI 2024), pp. 8048–8057. doi:10.24963/ijcai.2024/890.
Hao, S., Sukhbaatar, S., Su, D., Li, X., Hu, Z., Weston, J. and Tian, Y. (2025) 'Training large language models to reason in a continuous latent space', Conference on Language Modelling (COLM), arXiv:2412.06769.
Hidalgo, C.A. (2015) Why Information Grows: The Evolution of Order, from Atoms to Economies. New York: Basic Books.
Hong, S., Zhuge, M., Chen, J., Zheng, X., Cheng, Y., Zhang, C., Wang, J., Wang, Z., Yau, S.K.S., Lin, Z., Zhou, L., Ran, C., Xiao, L., Wu, C. and Schmidhuber, J. (2023) 'MetaGPT: Meta programming for a multi-agent collaborative framework', arXiv preprint, arXiv:2308.00352.
Jiang, L., Chai, Y., Li, M., Liu, M., Fok, R., Dziri, N., Tsvetkov, Y., Sap, M. and Choi, Y. (2025) 'Artificial Hivemind: The open-ended homogeneity of language models (and beyond)', Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS), Track on Datasets and Benchmarks.
Karpathy, A. (2026) autoresearch. GitHub repository. Available at: https://github.com/karpathy/autoresearch.
Lange, R.T., Imajuku, Y. and Cetin, E. (2025) 'ShinkaEvolve: Towards open-ended and sample-efficient program evolution', arXiv preprint, arXiv:2509.19349.
Lerchner, A. (2026) 'The Abstraction Fallacy: Why AI can simulate but not instantiate consciousness', Google DeepMind Technical Report.
Levin, M. (2019) 'The computational boundary of a "Self": Developmental bioelectricity drives multicellularity and scale-free cognition', Frontiers in Psychology, 10, 2688. doi:10.3389/fpsyg.2019.02688.
Ma, Y., Tsao, D. and Shum, H.-Y. (2022) 'On the principles of parsimony and self-consistency for the emergence of intelligence', Frontiers of Information Technology & Electronic Engineering, 23(9), pp. 1298–1323. doi:10.1631/FITEE.2200297.
Mieczkowski, E., Collins, K.M., Sucholutsky, I., Vélez, N. and Griffiths, T.L. (2026) 'Language model teams as distributed systems', arXiv preprint, arXiv:2603.12229.
Mikolov, T., Chen, K., Corrado, G. and Dean, J. (2013) 'Efficient estimation of word representations in vector space', Proceedings of the 1st International Conference on Learning Representations (ICLR 2013), Workshop Track, arXiv:1301.3781.
Moura, J. (2024) CrewAI: Framework for orchestrating role-playing, autonomous AI agents. GitHub repository. Available at: https://github.com/crewAIInc/crewAI.
Olsson, C., Elhage, N., Nanda, N., Joseph, N., DasSarma, N., Henighan, T., Mann, B. and Olah, C. (2022) 'In-context learning and induction heads', Transformer Circuits Thread, arXiv:2209.11895.
Pan, Z., Song, X., Wang, Y., Cao, R., Li, B., Li, Y. and Liu, H. (2025) 'Do code LLMs understand design patterns?', Proceedings of the LLM4Code Workshop, ICSE 2025, arXiv:2501.04835.
Rofin, M., Naghiyev, J. and Hahn, M. (2026) 'Understanding the emergence of seemingly useless features in next-token predictors', Proceedings of the 14th International Conference on Learning Representations (ICLR 2026), arXiv:2603.14087.
Tong, J., Li, M., Li, H., Yang, Y., Mou, Y., Ma, W., Xi, Z., Chen, H., Liu, X., Cheng, Q., Chen, M., Zhang, M., Ge, W., Guo, Q., Ying, T., Sun, T., Zheng, Y., Chen, X., Zhao, J., Ding, N., Huang, X., Jiang, Y. and Qiu, X. (2026) 'AI can learn scientific taste', arXiv preprint, arXiv:2603.14473.
von Oswald, J., Niklasson, E., Randazzo, E., Sacramento, J., Mordvintsev, A., Zhmoginov, A. and Vladymyrov, M. (2023) 'Transformers learn in-context by gradient descent', Proceedings of the 40th International Conference on Machine Learning (ICML), PMLR 202, pp. 35151-35174.
Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., Chi, E.H., Hashimoto, T., Vinyals, O., Liang, P., Dean, J. and Fedus, W. (2022) 'Emergent abilities of large language models', Transactions on Machine Learning Research.
Wigner, E.P. (1960) 'The unreasonable effectiveness of mathematics in the natural sciences', Communications on Pure and Applied Mathematics, 13(1), pp. 1–14. doi:10.1002/cpa.3160130102.
Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X. and Wang, C. (2023) 'AutoGen: Enabling next-gen LLM applications via multi-agent conversation', arXiv preprint, arXiv:2308.08155.
Yamada, Y., Lange, R.T., Lu, C., Hu, S., Lu, C., Foerster, J., Clune, J. and Ha, D. (2025) 'The AI Scientist-v2: Workshop-level automated scientific discovery via agentic tree search', arXiv preprint, arXiv:2504.08066.