Creating Reliable Systems from Unreliable Parts

The Fundamental Insight

In 1945, John von Neumann faced a critical challenge: building reliable computing machines without reliable components. His insight—that engineers must "learn how to design a reliable 40-stage machine with thousands of unreliable components"—remains central to AI engineering today.

Modern discussions often treat reliability as a problem to solve rather than a constraint to design around. Yet the structural nature of AI systems suggests otherwise: error is unavoidable, explanations are incomplete, and real-world environments always differ from training conditions. This isn't pessimistic—it's pragmatic design guidance.

The Historical Pattern

Building reliable systems from imperfect components recurs throughout technology history. ENIAC engineers faced constant vacuum tube and resistor failures, yet solved this through architectural redundancy and error-correction rather than waiting for perfect parts.

Similar patterns emerge in the SAGE air-defense system, which paired every computer with identical backups, and modern semiconductors, where "the most expensive mass-produced machine tool in history" achieves reliability through orchestration of specialized expertise rather than component perfection.

The Organisational Dimension

Reliability emerges from both technical and human structures. Bell Labs achievements required "a compilation of the ideas and inventions of individuals bound together with common purposes and complementary talents." The Skunk Works succeeded by maintaining "a small, intensely cohesive group" with tight feedback loops that caught errors before propagation.

Failure as Teacher

Unreliability generates valuable information. Failure analysis "reassemble[s] the whole into something greater than the sum of its broken parts," improving future designs. Don Norman's Swiss cheese model demonstrates that defensive layers—each imperfect—combine effectively when designed systematically.

A Personal Archaeology of Unreliability

1995: Natural Language as Approximation

An early NLP system for manufacturing facilities demonstrated practical wisdom: rather than creating a "smart" parser, the author collapsed unbounded input into bounded output spaces where errors could be caught quickly. Humans remained essential, not as failures, but as the meaning substrate itself.

Cardboard and Constraints

Control systems for cardboard manufacturing revealed that reliability requires orchestrating human judgment with deterministic optimization. Computers held physical boundaries; humans made desirable choices within those limits.

Telecommunications: When Software Cannot Save You

Distributed system failures couldn't be resolved algorithmically. Instead, monitoring systems checked constraints that should always hold. Statistical triggers identified anomalies; human response followed. Unreliability became legible rather than eliminated.

HR Systems: Defaults as Design

A candidate-ranking system exposed how UI design can mask underlying uncertainty. When defaults determine decisions due to volume constraints, system failures may align with user expectations—or create dangerous misalignment.

Computer Vision: Humans Augmented to Their Limit

Marathon runner detection at 85-95% accuracy seemed impressive until scaled: 5% error across thousands produces significant misidentification. Rather than pursuing perfect models, manual checking layers augmented AI rather than replacing humans.

Research revealed that cloud vision services evolve unpredictably with undocumented changes. Services aren't "plug-and-play"—developers face vendor lock-in due to inconsistent vocabularies and ontologies.

X-rays and Cognitive Load

Medical imaging work shifted focus from raw accuracy to presentation design. Accuracy varies across contexts; usefulness depends partly on failure patterns. Appropriate information presentation neither hides uncertainty nor overwhelms clinicians.

Software Evolution: Homeostatic Envelopes

Systems have stability ranges; push beyond them and they degrade. Well-tested functions break when dependencies change; calibrated models degrade with input distribution shifts. Reliability belongs to systems existing in time, not isolated components.

Smart Homes and Action Spaces

Smart home sensors, inference algorithms, and state spaces were all probabilistic. Yet available actions were limited: alert caregivers, adjust temperature, log anomalies. Internal complexity collapsed into smaller action spaces where humans could respond meaningfully.

The Recurring Pattern

Thirty years across multiple domains reveal consistent structures:

• Inherent unreliability exists at every layer: sensors, algorithms, humans, networks, organizations
• Space hierarchy: vast data spaces narrow to smaller decision spaces, then smallest action spaces
• Orchestration builds reliability: human layers, monitoring, constraint optimization, thresholds, and thoughtful UX create coherence despite component imperfection
• Uncertainty presentation matters: masking can help or harm depending on user mental models
• Systems evolve: today's working solution may fail tomorrow as environments and expectations change

The Physics of AI

These patterns suggest structural constraints governing AI systems. Nine constraints organize into three categories:

Adaptation Constraints address learning and change:

• Error is inherent; models approximate reality
• Interpretability is partial; explanations simplify
• Training differs from deployment; environments shift

Regulation Constraints address behavior boundaries:

• Human feedback defines objectives
• Resources (quotas, latency, cost) constrain feasibility
• Alignment is partial; metrics capture only fragments of intent
• Small changes cause large behavioral shifts

Resilience Constraints address system robustness:

• Entanglement: data, code, and humans interdepend
• Benchmarks expire as tasks and expectations evolve

These aren't optional design choices—they're structural features requiring deliberate engineering responses.

From Principles to Practices

Invest in experiment tracking: When error is inherent and environments drift, deployments become experiments. Infrastructure for tracking and learning is core capability.

Make observability standard: Partial interpretability and fragile robustness mean visibility into system behavior is essential.

Foster learning cultures: Many experiments "fail." Organizations succeeding with uncertainty maintain small, cohesive teams with tight feedback loops.

Define business metrics: Technical metrics are proxies. The durable measure is whether systems create value for actual humans.

Understand evaluator bias: Humans providing feedback shape systems. Their biases demand understanding.

The Silver Lining

This framework constrains methodology but not outcomes. The goal isn't eliminating unreliability—that's impossible—but orchestrating imperfect parts into useful wholes. This is achievable with effort.

ENIAC builders learned this. SAGE engineers learned it. Every semiconductor facility learns it anew. AI engineering must reflect and integrate these lessons.

Von Neumann's 1945 answer was yes—reliable systems can emerge from unreliable components. In 2026, that answer remains valid.

---

This work draws on research from the Deakin Applied Artificial Intelligence Initiative. The "Physics of AI" framework appears in a paper presented at CAIN 2026 (co-located with ICSE 2026).

---

References

[1] G. Dyson, Turing's Cathedral: The Origins of the Digital Universe. Pantheon Books, 2012.

[2] S. Barnett, A. Pasquini, S. Kurniawan, S. Sivasothy, R. Hill, and R. Vasa, "The Physics of AI," in Proc. 5th IEEE/ACM International Conference on AI Engineering - Software Engineering for AI (CAIN '26), Rio de Janeiro, Brazil, Apr. 2026.

[3] M. M. Waldrop, The Dream Machine: J.C.R. Licklider and the Revolution That Made Computing Personal. Viking, 2001.

[4] T. Kidder, The Soul of a New Machine. Little, Brown and Company, 1981.

[5] C. Miller, Chip War: The Fight for the World's Most Critical Technology. Scribner, 2022.

[6] J. Gertner, The Idea Factory: Bell Labs and the Great Age of American Innovation. Penguin Press, 2012.

[7] B. R. Rich and L. Janos, Skunk Works: A Personal Memoir of My Years at Lockheed. Little, Brown and Company, 1994.

[8] G. P. Zachary, Showstopper! The Breakneck Race to Create Windows NT and the Next Generation at Microsoft. Free Press, 1994.

[9] H. Petroski, To Engineer Is Human: The Role of Failure in Successful Design. St. Martin's Press, 1985.

[10] D. A. Norman, The Design of Everyday Things, Revised and Expanded Edition. Basic Books, 2013.

[11] A. Cummaudo, R. Vasa, J. Grundy, M. Abdelrazek, and A. Cain, "Losing Confidence in Quality: Unspoken Evolution of Computer Vision Services," in Proc. 35th IEEE International Conference on Software Maintenance and Evolution (ICSME '19), Cleveland, OH, USA, 2019, pp. 333-342.

[12] A. Cummaudo, S. Barnett, R. Vasa, J. Grundy, and M. Abdelrazek, "Beware the Evolving 'Intelligent' Web Service! An Integration Architecture Tactic to Guard AI-First Components," in Proc. 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '20), Virtual Event, USA, 2020, pp. 269-280.

[13] A. Cummaudo, S. Barnett, R. Vasa, and J. Grundy, "Threshy: Supporting Safe Usage of Intelligent Web Services," in Proc. 28th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE '20), Virtual Event, USA, 2020, pp. 1645-1649.