Eddie Obeng holds a special place in the Architect Tomorrow story. His concept of “the world after midnight”, the idea that the pace of change overtook our ability to learn and adapt sometime around the early 1990s, was a foundational inspiration for this podcast and community. So it was a genuine privilege to finally sit down with him on location in his office for a conversation that ranged across engineering, ecosystems, governance, sustainability, and what it actually takes to make progress in a complex world.

For those unfamiliar with Eddie: he’s an engineer turned educator turned author turned sense-maker. Chemical and biochemical engineering degrees, a PhD, an MBA, ten books (including Financial Times bestsellers), a TED talk, work with roughly 70% of the FT 100 over his career, and the creation of Qube, a collaborative digital platform. He’s also one of the fastest talkers you’ll ever encounter, which suits me just fine as I tend to run at 1.25x speed myself!

What follows are the ideas from our conversation that I think matter most for architects, systems thinkers, and anyone trying to build something better.

A word of context before we dive in: Eddie is a provocateur, and deliberately so. He takes positions to their extremes to force you to examine your assumptions. Some of what follows, particularly on governance and the role of government, I found myself pushing back on in the conversation and still wrestle with now. That’s the point. I don’t think you have to agree with Eddie on everything to find his thinking enormously valuable. Regular readers will know that this newsletter has consistently argued for governance as an innovation enabler. But I also believe that uncomfortable conversations sharpen thinking. Take what challenges you. Discard what doesn’t. And ideally, let me know where you land in the comments.

The full podcast is now available here: https://www.youtube.com/watch?v=i2qbwlaDMeM

The World Already Changed. You Just Didn’t Notice.

The “world after midnight” concept sounds dramatic, but Eddie’s explanation is rooted in straightforward observation. Working with telecoms and banking clients in the early 1990s, he noticed organisations doing sensible things and getting nonsensical results:

“They had a mindset of how the world changed. They had a pace to themselves: quarterly, monthly, annual. That’s how they saw the world. Meanwhile, everything was connecting up. There were more people on the planet. They were moving into cities. The technology was going faster and faster.”

Plot the pace of change (exponential, self-reinforcing) against our pace of learning (constrained by meetings, hierarchies, attention spans), and you get a crossing point. That crossing point was midnight:

“The world had already flipped. But you hadn’t noticed because no one rang a bell. Hence, midnight. We were asleep.”

He first presented this at Ashridge Business School in 1992. At the time, he says, it was “basically bonkers.” Over three decades later, it feels more relevant than ever. Every organisation I work with is grappling with this gap, and the emergence of generative AI has arguably widened it further.

The Wildebeest and the Immune System

I shared my concept of the “corporate immune system”, the idea that organisations have a natural defence mechanism that pushes out ideas that are too radical. Eddie appreciated the thinking but offered a characteristically blunter metaphor:

“Once the wildebeest start running, they’re going in a direction. You go, ‘I think we should go this way.’ They nudge you back. You do it again and this time they just knock you over and trample you on and move on. I think it’s the momentum of all the old ideas, heuristics, processes, and KPIs. It just flattens you as you go.”

I laughed, and then stopped laughing, because it’s painfully accurate. The wildebeest image captures something the immune system metaphor perhaps doesn’t: it’s not a sophisticated defence mechanism. It’s just momentum. There’s no malice in it. The herd doesn’t decide to trample you. It simply can’t change direction fast enough.

This connects directly to the challenges we’ve discussed on this podcast around AI adoption. Organisations are applying old transformation patterns (waterfall planning, fixed target states, linear business cases) to an immature emerging technology (that exhibits different traits due to its non-determinism). The wildebeest are running. The question is whether architects can find ways to redirect the herd or whether they’ll keep getting trampled.

Events Evaporate, Patterns Persist

If there was one phrase from our conversation that I’d print on the wall of every architecture practice, it’s this:

“The best way of dealing with the situation is not to deal with the events which come at you, but to spend some time looking at the underlying patterns and underlying causes.”

Eddie went further: “Even in a complex system, there are rarely ever more than half a dozen things you really should focus on. And if you focus on those, you always have all the time in the world. If you focus on all the events, events evaporate, patterns persist.”

This is architectural thinking at its most fundamental. And yet how many architecture teams spend their time chasing events? A new AI model launches, a competitor makes an announcement, a vendor pitches a shiny platform, and the whole team pivots. Meanwhile, the persistent patterns (data fragmentation, organisational silos, skills gaps, sustainability debt) go unaddressed.

Ron Kersic talked about “incremental truth” in his episode on futuring architectures. Jenny Wilson brought the lens of evolutionary thinking from her zoology background. Eddie’s “events evaporate, patterns persist” is the same insight from yet another angle. The best architects I know have internalised this. They don’t ignore events, but they don’t let events drive their strategy either.

Dynamic Stability: The Spinning Top, Not the Vase

One of the most thought-provoking moments came when Eddie challenged the common understanding of sustainability. His mother was an environmental biologist and parasitologist, so he grew up with sustainability meaning ecosystem dynamics, not corporate ESG reports:

“When people say sustainability, I smile because I know they haven’t thought through what that actually is. They all imagine it stabilising flat on the table.”

He described three types of stability: flat on a table (static equilibrium), balanced in a bowl (dynamic but self-correcting), and balanced on a mountaintop (unstable, any perturbation is catastrophic). Most organisations aim for the first. Most sustainability strategies assume the first. But real systems, the ones that endure, work like the second:

“What’s more useful is what I call dynamic stability, which is the spinning top. You hit a spinning top and it’s rotating but stays upright. It’s the balance. And the spinning top in the bowl. Fantastic. Because it can move around.”

The implications for architecture are profound. If you’re designing for a static target state, you’re designing for a world that doesn’t exist. You need to design for dynamic stability: systems that maintain their essential purpose whilst continuously adapting to changing conditions. This resonates with the cone of possibilities framework I’ve been developing, which explicitly maps multiple futures rather than trying to predict a single one.

It also has direct implications for how we think about AI sustainability. As I discussed at HM Treasury’s ID25 event last year, the brute-force approach to AI is inherently fragile. Massive centralised infrastructure, enormous energy consumption, and winner-takes-all economics are the opposite of dynamic stability. A more distributed, right-sized, and adaptable approach to AI infrastructure would be far more resilient.

The Problem with Governance (and Government)

Here’s where the conversation got properly spicy and where I want to be transparent about where Eddie and I diverge. Eddie holds a view that essentially all government intervention distorts markets and impoverishes people, a position I pushed back on but found fascinating to explore. His argument centres on the idea that intervention, however well-intentioned, always steals resources from the working system:

“You put guardrails and you’re stealing from someone. Who are you stealing from?”

He illustrated this with his experience designing the UK’s Sparks innovation investment model: government announces innovation grants for an emerging sector, people who aren’t in that sector but want the money write proposals to get it, and end up competing with (and sometimes destroying) the very innovators the grants were meant to help.

I’ll be honest: I struggle with this in its purest form. I think governance (as distinct from government) plays a vital enabling role, and Selena Evans made this case powerfully in our previous episode: “Governance unlocks your innovation engine.” But Eddie’s point about scale and transparency is one I find much harder to argue with:

“It’s not the size which is important. It’s the scale and transparency. If you can see what people are doing, they tend not to be able to cheat so much.”

The architectural lesson here isn’t about adopting Eddie’s political philosophy wholesale. It’s about designing governance that is proportionate, transparent, and scaled appropriately. Bad governance, as Selena also noted, “slows things down… in the way that organisations practise it very defensively and with a frame of risk aversion at all costs.” Good governance is invisible, enabling, and honest about trade-offs. I still believe governance is essential, particularly for AI deployment and sustainability. But Eddie’s challenge on scale and transparency is one I think we should take seriously rather than dismiss. As I often say: give me the freedom of a tight brief.

All the Problems of the World Are Trivial

This was the claim that made me laugh out loud. But Eddie backed it up with a story.

Trying to sell Qube to the Scottish government, he was drawing a diagram of commuting journeys and their carbon impact. He noticed that every location in the journey (home, car, train, office) had space heating. He recalled skiing at minus 20 without burning a single carbon. The solution?

“What if I dress properly? You solve climate change by just persuading everyone to dress differently. All you need is a celebrity or a designer who designs the really stunning fashion.”

Now, I don’t think this literally solves climate change, and I suspect most of you reading this don’t either. But the underlying point is incisive: problems that should be engineering and design challenges get handed to people who worry, then to politicians, then to blame exercises, and the actual problem solvers never get in the room. As Eddie put it:

“The whole conversation at a global level didn’t involve engineers, architects, designers, problem solvers.”

This resonates deeply with the architect’s predicament. We’re often brought in after the strategic decisions have been made, asked to implement rather than shape. Selena Evans and Darryl Carr both argued in our human-centred AI episode that architects should lean in to policy-making and civic architecture. Eddie’s provocation takes this further: if you’re a problem solver, stop waiting to be invited. The problems are trivial. It’s the politics, fashion, and “cocktails” (his word) that make them hard.

The Human Side: Issue, Data, Question, Build

For all the big-picture thinking, Eddie’s most practically useful advice was about working with people. He described one of his PETs (Performance Enhancement Tools) called “Issue, Data, Question, Build”:

“If somebody suggests something to you and you haven’t come up with it on your own, your brain basically panics. It treats it as a change. Something is going to eat me. But questions work. If you ask someone a question, it triggers a different part and it goes after it.”

The technique: raise an issue the other person recognises, provide concrete data, ask a question (triggering their brain’s problem-solving mode), then shut up and let them engage. Once they start answering, join the dialogue. Don’t get precious about ownership.

“That allows other people to own your idea.And to come up with something creative. Truly collaborate.”

This is gold for architects who constantly need to influence without authority. We’ve talked about this challenge across many episodes, and Eddie’s framework gives it a practical shape. The brain falls in love with its own answers far more than being told what to do.

Cybernetics and Collective Intelligence

Selena Evans asked me to put a question to Eddie about his naturally cybernetic approach and sustainable adaptability. His response was characteristically honest about human limitations:

“As human beings, we can’t deal with as many variables as natural systems. So we pretend we’re building complex systems and they’re really quite simple systems. And then we become the god of our own simple system and think it can resolve everything, which of course it can’t.”

His solution isn’t better central planning. It’s recognising that semi-connected, interdependent individuals working together can handle complexity that no individual or hierarchy ever could:

“We will never be able to deal with the complexity as individuals which we can deal with by being disconnected and semi-connected individuals.”

This is why he built Qube. It’s why he’s passionate about collaboration tools that let introverts contribute (they “have superpowers” but “won’t say a word” in traditional meetings). And it’s why he believes organisations that rely on pyramid structures and bottleneck decision-making through a handful of senior leaders are fundamentally mismatched to the world after midnight.

Don’t Screw It Up

Eddie closed with advice that was both simple and profound:

“The first most important thing is you. You have to decide whether your goal is to contribute. And if you’ve decided to contribute, then it gets easy. Because from here on, everything you do, you know is the right thing to do.”

And then, with characteristic Eddie flair:

“We’re all here. It’s an adventure. The funny part of this adventure is for all of us, it probably started several billion years ago. Because life never ends. It just continues. You get to this point after billions of years and your chance to do something which can be great. Don’t screw it up.”

What This Means for Architects

This conversation reinforced several themes that keep emerging across the Architect Tomorrow community:

Stop chasing events. The AI announcements, the vendor pitches, the market panics: these are events. They evaporate. Focus on the half dozen patterns that actually matter for your organisation. As we discussed in the AI principles piece (https://www.linkedin.com/pulse/practical-pragmatic-ai-principles-autumn-oliver-cronk-jkwfe/), demanding evidence over enthusiasm and mapping reality over marketing are patterns that persist.

Design for dynamic stability. Static target states are a fiction. Whether you’re designing AI architectures, organisational structures, or governance frameworks, build for adaptability. The spinning top in the bowl, not the vase on the table.

Work with human psychology, not against it. Architects influence through ideas, not authority. Eddie’s Issue, Data, Question, Build framework is one practical technique, but the principle is broader: help people fall in love with shared answers rather than trying to impose your own.

Get in the room earlier. If problems are trivial but politics makes them hard, architects need to be involved in shaping strategy, not just implementing it. Lean in to policy, governance design, and ecosystem thinking. As we’ve explored across multiple episodes, this is where architects add irreplaceable value.

Decide to contribute. Everything else follows from this. If you’ve made that decision, the wildebeest might still trample you occasionally, but at least you’ll know which direction you were trying to go.

Eddie Obeng’s work was the spark that lit Architect Tomorrow. This conversation reminded me why. In a world after midnight, where the pace of change has long since outstripped our ability to comprehend it, we need sense-makers, pattern-finders, and people brave enough to ask uncomfortable questions. That’s what this community is about. That’s what architects do.

As Eddie would say: it’s an adventure. Don’t screw it up.

You can find Eddie’s PETs TED talk, his (Performance Enhancement Tools), books, and more by searching for Eddie Obeng online, or follow him directly on here.

More information about QUBE: https://home.qube.cc/

Get started on QUBE: https://home.qube.cc/register

There was a nice bit of symmetry to this recording. Back in February 2023, we had Whynde on the podcast with Lisa on the panel, discussing Whynde’s book Strategy to Reality. Three years later, the tables have turned and we are asking Lisa about her book instead!

Embedded LinkedIn content: view it on the original article

Blog to Book and Corridors to Chapters

Lisa’s book didn’t start as a book. It started as a decade of blogging, 156 posts, over 100,000 words, driven by a simple instinct to make sense of what she was seeing inside transformation programmes:

“That blog came pre-AI. It was just my responses to what I was seeing happening either at work or on LinkedIn. Stuff that Whynde was posting about was triggering me to think about what do I really think about that. And I just was blogging, getting my thoughts down, publishing it on the internet and seeing what conversations came as a result.”

What emerged from that body of work was a pattern. The conversations that mattered, the ones that determined whether transformation would succeed or quietly fade, were happening in corridors not in technical design sessions:

“I thought there is something in all this lived experience which I can bring together and maybe help everybody in the technology world of enterprise architecture think a bit differently, think about being more human and more honest. Because that was conversations I was hearing in the corridors but I wasn’t hearing in the technical discussions.”

That observation alone should give any architect pause. How much of the signal that matters to our work is being picked up in the formal forums, and how much is being lost because we’re not in the right conversations?

The Five Lenses: Permission to Step Back

The heart of the book is the Five Lenses of Transformation: Reflect, Reimagine, Reframe, Rewire and Reconnect. Lisa was emphatic that these are not another framework to follow:

“It’s not a framework. It’s not a process. It’s stepping back sometimes as architects. Stepping back from the technical diagrams, stepping back from the technical conversations and reflecting on why we are where we are. We’ve got a legacy estate for a reason. People that came before us have made decisions based on the context of the time.”

That last point is worth sitting with. We spend a great deal of time complaining about legacy estates without acknowledging that someone, probably someone quite smart, made those choices with the best information available at the time. Reflection means understanding the context before reaching for the solution.

Of the five lenses, Lisa identifies Rewire as the core: “That’s where we all spend most of our time. We are rewiring people. We’re rewiring processes. We’re rewiring technology.” But it is Reconnect that she considers the most important, and the one most often left unread:

“We can talk about AI, we can talk about cloud, we can talk about blockchain. I remember we’ve all forgotten that, but that was a hype thing. But why are we doing all of this stuff? We’re doing it for people. And it’s actually what shows up in that technology and how we reconnect with the people we’re serving.”

Whynde distilled the book’s contribution sharply: “This book brings us back to two things: intent and human.”

Don’t Learn Everything. Bring People In.

One of the most practical challenges to come out of the conversation was Lisa’s pushback on the assumption that architects need to develop human-centred skills themselves. Instead, she argued for broadening who sits at the table:

“I’m not saying we need to all develop the skills to be more human-centred. I’m saying make sure you’re opening your ears and your eyes to people that can bring that specialism into your teams.”

She spoke about a behavioural researcher she worked with at WPP who transformed her own thinking about technology deployment:

“She wasn’t a researcher of the business or technology. It was very much about human behaviours. She really demonstrated for me the power of a day in the life, actually getting out into the organisation, watching what people are doing, listening to what people are saying, what they were thinking before they open their laptop. You can’t capture that in your requirements document.”

This echoes Jenny Wilson’s point from a previous Architect Tomorrow episode about architects being enablers, not gatekeepers. The question is not whether architects need to become behavioural scientists. It is whether architecture teams are diverse enough to include the perspectives that matter. Lisa’s answer is direct: “Even if you just do that once a year, just get out there and see what people are experiencing in their work. I think you think about your technology challenges differently. I know I did.”

The Unsaid: What Lurks Below the Surface

Whynde picked up on a line from the book that deserves to become a design principle in its own right: “Lisa says, ‘How change feels to people matters as much as what it does.’ I don’t think we spend nearly enough time, and also too late, focusing on that change.”

She identified two structural failures that let the emotional and political dimensions of transformation derail progress. The first is the absence of genuine spaces for dialogue:

“When we don’t create real authentic forums and spaces for people to dialogue and ask questions and voice what they’re thinking, all that stuff is lurking below the surface. Sometimes change management even when we do it, it feels very mechanical.”

The second is more systemic: “The organisational structures and motivations of leaders, which can create really siloed political decision-making.”

That word “mechanical” is telling. We have got quite good at the theatre of change management: the communications plan, the stakeholder map, the readiness assessment. But if the forums are not authentic, if people cannot say what they actually think, then it is all surface.

Designing for the Ecosystem, Not Just the Enterprise

The conversation took a significant turn when we moved beyond the enterprise boundary. Lisa made an observation that connects to something I have been working on with Pragmatic Ecosystems and Dynamic Ecosystem Architecture:

“We’ve often been at the forefront of designing the new organisations, what we’re going to outsource, which platforms we’re going to bring in. That is all part of an ecosystem. And actually it’s not about designing for our organisation, it’s about designing for the ecosystem that we sit within and being sure of the consequences not just for us, our teams, our processes and our systems, but also for the teams, processes and systems of the organisations that are part of our supply chain. Because they’re often thought of last.”

This is a critical insight. Too much of our architectural thinking stops at the organisation boundary. The decisions we make about platforms, outsourcing, and integration ripple through partner organisations and supply chains in ways we rarely model or consider. As I noted during the discussion, Ron Kersic put it well in our Futuring Architectures episode: “It’s not your ecosystem. You cannot build it. You cannot design it. You can design your position.”

Lisa connected this to the practical realities of post-implementation reviews: “If you really look back over the last 20 or 30 years of the work you’ve been doing, you will recognise those meetings that happened, those post-implementation reviews that would have highlighted the engagement of stakeholders too late. And it’s often those ecosystem stakeholders that we’ve involved too late.”

Whynde extended the ecosystem concept into business architecture constructs, noting that value networks and value streams can readily accommodate sustainability dimensions: “We can use the same construct and map on it materials and energy. Circular economy thinking.” This is exactly the kind of thinking we need more of, reusing architectural tools we already have rather than inventing new ones.

The Natural Ecosystem We Conveniently Ignore

I pushed the conversation further to address the ecosystem we most often overlook. Our worlds are increasingly digitised, virtualised and abstracted from the physical world. But fundamentally, we are all part of the natural ecosystem. We are going to start facing into some hard limits to technological progress: there are only so many data centres we can build, and only so much energy we can distribute to power them.

Ecosystem thinking in its fullest sense means considering not just the business ecosystem but the societal and environmental ecosystems that underpin everything. As architects, we are collectively making choices and decisions that impact sustainability in its broadest sense. The ripple effects of individual projects may seem small, but in aggregate they matter enormously.

The Ironies of Automation: Sleepwalking Risks in Practice

An audience question (from Charles Betz of Forrester, who will appear himself on the podcast when the Twin Cities Business Architecture Forum episode goes live) brought the conversation sharply into the present. Referencing Lisanne Bainbridge’s 1983 paper Ironies of Automation (which inspired a recent newsletter), which argued that humans left “in the loop” to monitor automated systems will inevitably lose the context needed to intervene when those systems fail, the questioner asked whether business architecture has started to address this.

Lisa’s response was characteristically direct:

“Enterprise architects have a role to play in slowing down maybe some of the hype and potential that AI can deliver to allow that human side to catch up. The figures this morning: 1.5 trillion being spent on AI. How much of that is going into the technology side and how much is going into that human capability side? I would say it’s probably 98% on the technology and 2% on the human side. I think we’ve got to correct that.”

This connects directly to the sleepwalking risks we have been exploring on this podcast. Confirmation fatigue. Capability degradation. The quiet erosion of skills and engagement when you take the interesting work out and leave people staring at a production line looking for anomalies. As I noted during the discussion: how interesting are these roles when you take the friction out? There was a recent post about software engineering that captured this perfectly: is it really more fulfilling to debug AI-generated code than to get the dopamine hit from solving the problem yourself?

Whynde brought the macro perspective: “There’s a lot of things we can do and this conversation is so focused at a micro level. We need to come up and say, macro, does this make sense where we’re going?”

That is exactly the right question.

Two Books, One Mission

When asked to compare Whatever Next? and Strategy to Reality, Lisa and Whynde positioned them as companions:

Lisa: “My book is more a lived experience. It’s someone’s reflections on living through the last 30 years of transformation. Whynde’s would be more of the go-to book for business architecture in strategy execution.”

Whynde’s response was simply: “They’re companions.”

They are. And the architecture community is richer for having both.

The Path Forward

Lisa summed up the signal that architects need to tune into:

“We’re enterprise architects. We join the dots. We see things that other people in the organisation don’t see because we have the privilege to work across the silos. We are also in a position where we can go deeper. We can pick up the signals that maybe others aren’t picking up because we’re not as siloed in our thinking. The real message is to turn the volume up on those signals, those hunches, that intuition that you’re getting and pay more attention to that.”

The question for all of us is whether we are listening for the right signals or just the loudest ones. The corridors as much as the ARBs. The supply chain partners as much as the internal stakeholders. The natural ecosystem as much as the business ecosystem.

Lisa’s book gives us five lenses to see more clearly. It is up to us to actually look.

Lisa Woodall is the author of Whatever Next? Making Transformation More Human, More Honest and More Likely to Stick. Find the book, the Five Lenses of Transformation and Lisa’s blog at whatevernextbook.com

Whynde Kuehn is the author of Strategy to Reality and Founder of S2E Transformation. Find her work at strategyintoreality.com and s2etransformation.com and her book interview here.

Oliver Cronk is Founder of Cronk Advisory: cronkadvisory.com a fractional CTO and chief architect consultancy specialising in regenerative and sustainable architecture, innovation and AI. Oliver is also the founder of Architect Tomorrow.

This episode was recorded live at Chief Architect Network London 2026. Find out more about CAN and how to apply to join at chiefarchitectnetwork.com

AI use disclosure: This newsletter was written by Oliver with assistance from a single Claude 4.6 Opus prompt to pull out highlighted parts of the transcript. With human review and editing of the output.

See also related previous editions of the newsletter / podcast:

• Futuring Architectures with Ron Kersic: ecosystem thinking and the cone of possibilities
• From Zoology to Systems Thinking with Jenny Wilson: architects as enablers
• Architecting Human-Centered AI with Selena Evans and Darryl Carr
• Are We Sleepwalking into Tomorrow’s AI Challenges?
• Practical and Pragmatic AI Principles for AI Autumn

I have come across three interesting patterns in older automation/economic literature that could be significant to how viable and sustainable agentic AI / automation programmes are. Significant food for thought that I welcome push back and discussion on!

We already have a lot of the answers but they are gathering dust!

Thanks to members of the community (in particular Selena Evans) pointing me at some excellent papers (particularly on cybernetics) from the 70s and 80 and the Ironies of Automation getting attention; I wondered what else might be out there. What follows is a set of observations: three old fallacies from automation, economics, and human factors that shape the non-happy-path of using enterprise AI to drive automation.

What are the AI sleepwalking risks?

As a quick reminder / orientation, here is how the sleepwalking risks sit together when you draw the connections between them. Keep this context in mind as we walk through the fallacies in a moment.

Enterprise / organisational sleepwalking (how AI reshapes institutions, markets, and competitive / ecosystem dynamics):

• AI Arms Race
• (Greater) Digital Divide
• Asymmetrical Overload (AI generated volumes can easily exceed human team capacity)
• IP & Knowledge Leakage (sensitive org material can end up in AI models)

Personal sleepwalking (how AI reshapes individual cognition, behaviour, and social experience):

• Confirmation / Augmentation Fatigue (unreliable human review)
• Relevance Degradation (AI causes expertise loss)
• Tech Addiction
• Impersonation (AI content indistinguishable from human)
• Human vs Bot (AI personas easily confused with humans)

Figure 1 - AI Sleepwalking Risks V2, some of the lines probably need tweaking!

Overview of diagram - nine sleepwalking risks split into enterprise (top) and personal (bottom). The critical channel runs vertically on the right, where Asymmetrical Overload becomes Confirmation Fatigue: the precise seam where Fallacy 2 bleeds into Fallacy 3. Relevance Degradation sits as the hub where two reinforcing loops meet.

The three fallacies

• The Substitution Fallacy. That automation cleanly replaces human work. Sits uncomfortably with Bainbridge (1983) and the human factors literature.
• The Productivity Fallacy. That automation investment yields measurable, proportionate productivity gains on a forecastable timeline. Sits uncomfortably with Solow’s 1987 observation and forty years of subsequent IT productivity research.
• The Oversight Fallacy. That a human reviewer can meaningfully supervise an automated system whose outputs they did not generate. Sits uncomfortably with Mackworth on vigilance and the automation bias research that followed.

Fallacy 1: The Substitution Fallacy

Automation replaces human work cleanly.

Lisanne Bainbridge’s paper “Ironies of Automation” (Automatica, 1983) outlines that “the designer who tries to eliminate the operator still leaves the operator to do the tasks which the designer cannot think how to automate.” The residue that remains is rarely a simplified version of the original job. It tends to be whatever resisted automation: the edge cases, the ambiguous judgements, the abnormal conditions. And it gets handed to a human who has lost the routine practice that originally built their competence.

Several of Bainbridge’s observations read differently in an agentic AI context than they might have a few years ago:

• Manual skills decay under (mostly) monitoring. A reviewer who rarely produces the underlying output loses the ability to evaluate it. (this one I have felt a lot personally and as a result my use of AI have changed in the last few weeks / months)
• Long-term knowledge requires frequent use. Theoretical training without practical exercise does not stick.
• Working context takes time to build. A human parachuted into an AI-generated decision often lacks the situational awareness that made the original operator effective.
• The most successful automated systems may need the greatest investment in human training. Precisely because intervention is rare, the competence required when it matters is fragile

Note I am not claiming automation cannot replace tasks. It clearly can, and often does so satisfactorily. The assumption worth questioning is whether that replacement is clean? My experience suggests it leaves issues that are less visible than the tasks replaced, harder to specify, and often under-invested in because it falls outside both the original role and the original business case.

A possible architectural response: treat the human residue as a first-class design concern rather than an afterthought. More controlled architecture patterns seems to help, because keeping the AI’s scope narrow keeps the residue legible. Agentic architectures tend to scatter the residue across a larger surface area, which could make it harder to support or govern?

Another possible response (anti-pattern?): Human oversight is too hard to sustain reliably so we won’t bother with it. Probably a high risk strategy but sadly I suspect many architects drunk on AI vendor kool-aid (or organisations that fired the whole team doing the work manually) may be tempted to go this way?

Fallacy 2: The Productivity Fallacy

Automation investment yields measurable, proportionate productivity gains on a forecastable timeline.

In July 1987, Robert Solow wrote a book review in the New York Times that produced a fascinating quote that I am amazed I’ve not seen before:

“You can see the computer age everywhere but in the productivity statistics.”

By the late 1980s, US firms had invested heavily in IT for nearly two decades, and measured productivity growth had slowed over the same period. The Solow paradox became the defining puzzle of technology economics for roughly a decade. When productivity growth eventually picked up in the late 1990s, it arrived considerably later than originally forecast, concentrated in a minority of sectors, and with gains captured disproportionately by a narrow band of frontier firms.

Erik Brynjolfsson and colleagues have more recently attempted to explain why. Their “productivity J-curve” framework argues that general-purpose technologies require large, mostly-invisible complementary investments before measured productivity improves. Organisations have to redesign processes, develop new skills, restructure around new capabilities, and sometimes discard what used to work. During that period measured productivity can be worse, because the intangible investment costs are counted but the benefits have not yet materialised. Eventually the curve turns upward, but the lag has historically been long.

A few recent studies are worth considering soberly alongside this. Probably not wise to lean heavily too on any single one, and the picture they collectively paint is still forming:

• Brynjolfsson, Li and Raymond (2023) reported generative AI improved customer service agent productivity by roughly 14%, concentrated mostly in less-experienced workers. Experienced workers saw relatively small gains.
• The Dell’Acqua et al. (2023) “jagged frontier” study with BCG consultants found material performance improvements on tasks “within the AI frontier”. However a measurable decline on tasks that looked similar but fell outside it and consultants often could not tell which were which.
• Daron Acemoglu’s 2024 paper “The Simple Macroeconomics of AI” offers a notably lower aggregate productivity estimate for AI over the coming decade than most consulting forecasts.

These studies have been contested, as you would expect with anything touching AI economics right now. Taken together though, the pattern they suggest is that AI productivity gains may be real but concentrated, uneven, and probably smaller and slower than many deployment cases assume.

Where this matters architecturally / from a governance standpoint is the pressure that unrealistic productivity targets place on design decisions.

When the business case demands 30% productivity gain in eighteen months, architects tend to get pushed toward:

• Visible gains over substantive ones. Dashboards, co-pilots, and demos that satisfy the quarterly review rather than process redesign (often less sexy or visible behind the scenes stuff) that would pay off over five years.
• Coverage over quality. Broad rather than careful deployments, because narrow high-quality deployments take longer to show in aggregate numbers.
• Thin oversight over deep oversight. Review processes sized to the productivity target rather than to the risk profile of the task. This is where Fallacy 2 starts to link into Fallacy 3.

A possible architectural response: plan for the J-curve. Make the intangible complementary investments (skills, process redesign, data quality, governance maturity) part of the delivery plan rather than deferrable. Resist the temptation to size review capacity to productivity targets rather than risk. And be candid with steering committees that the historical record does not always support the pace of gain currently being forecast.

Fallacy 3: The Oversight Fallacy

A human reviewer can meaningfully supervise an automated system whose outputs they did not generate. (very much links to Confirmation Fatigue and relevance degradation).

Bainbridge, put this more bluntly than I would:

“If the computer is being used to make the decisions because human judgement and intuitive reasoning are not adequate in this context, then which of the decisions is to be accepted? The human monitor has been given an impossible task.”

This is confirmation fatigue (or augmentation fatigue) stated in a reasonably rigorous form. My sense is that it is not primarily a motivation problem, and not primarily a training problem. It looks more like a structural feature of how review work sits against automated output volumes.

A few mechanisms seem to feed it, each with different architectural implications:

1. The vigilance decrement. Mackworth (1950) showed that human attention on rare-event signals tends to collapse within about 30 minutes. Review workflows that assume sustained attention over long shifts are building on a shaky premise.
2. Automation bias. Decades of research in aviation, medicine, and finance suggest humans systematically over-trust automated outputs even when contradicting evidence is visible. Modern LLMs produce fluently authoritative text, which seems likely to amplify the pull toward agreement.
3. The competence gap. The skills needed to review an output often overlap heavily with the skills needed to produce it. If AI automates the junior work that builds senior judgement, the reviewer population may become less able to detect confidently-wrong outputs.
4. Context poverty. Bainbridge observed experienced operators arrived thirty minutes early to build situational awareness. A reviewer handed an AI-generated decision sees the conclusion without that substrate of context. Explanation traces and chain-of-thought transcripts are not quite the same thing.
5. Automation camouflage. Automated systems can compensate against small deviations until the trend is beyond recovery. AI outputs that are wrong in subtle, internally-consistent ways may be particularly hard to detect. The Post Office Horizon scandal is a sobering point of reference: institutional trust in an automated system outlasted contradicting human evidence for over two decades.
6. The Ephrath effect. Ephrath (1980) found system performance sometimes worsened with computer aiding, because the operator made the decision anyway and checking the machine added load. The “I asked Copilot, then rewrote it from scratch” experience may be this effect rediscovered.

The usual mitigations have specific weaknesses I would flag rather than assume solved. “Human-in-the-loop” has become so universal it has lost some meaning; when a reviewer processes 200 outputs an hour against a target that assumes 200 outputs an hour, they are not obviously in any loop in a meaningful sense. Madeleine Clare Elish’s “moral crumple zone” research argues that the human in such arrangements absorbs moral and legal liability for failures the system’s design made hard to detect, which is a framing I find hard to argue with given how opaque many AI systems are.

LLM-as-judge evaluations are useful in development and high-throughput operational checks (they were used in building InferESG) but I would flag as a hypothesis that they may be vulnerable to recursive complacency: judge and candidate often share training data, tokenisation, and architectural biases, so the independence assumption is not obviously safe. This is a known unknown rather than a settled criticism, and worth scrutiny.

A possible architectural response: reserve agentic, thin-oversight architectures for lower-consequence tasks where crumple-zone dynamics are less acute. Use more controlled architectures where consequences are meaningful (yet more reinforcement for my recommendation of the use of deterministic tech to co-ordinate AI for highly regulated domains). And be careful about calling something human-in-the-loop unless the human plausibly has time, authority, and competence. If one or more of those is missing, it is probably more honest to call it a approval ceremony than a mechanism of oversight?

How the Fallacies Seem to Compound

If the three fallacies were independent they would be serious but manageable. But I think they might reinforce each other:

The substitution fallacy leaves an under-supported remainder of human work: edge cases, ambiguous judgements, oversight tasks.

The productivity fallacy then generates pressure for those residual tasks to be performed faster and cheaper, because forecast gains have not arrived on schedule and steering committees need evidence of progress.

Under that pressure the oversight fallacy becomes acute. Review capacity is sized to the productivity target rather than to the risk. Reviewers get less time, less context, and less authority than genuine oversight requires. A ceremonial version of human-in-the-loop takes over, which looks exactly like governance from the outside.

This is where the sleepwalking diagram becomes useful. The compounding pattern maps fairly directly onto the clustered structure: productivity pressure sits in the systemic cluster at the top, feeds through Asymmetrical Overload, and lands in the cognitive cluster where Confirmation Fatigue and Relevance Degradation start reinforcing each other.

The two feedback loops anchored on Relevance Degradation are where an organisation stops being able to self-correct; skills atrophy makes ceremonial review feel acceptable, ceremonial review removes the practice that would have maintained skills, and the loop tightens.

That makes Relevance Degradation something like the load-bearing wall in this structure. If it holds, the other risks remain manageable. If it goes, a lot goes with it.

I am not claiming this is the only way to read the interactions. Others may weight things differently. But it suggests that interventions targeting skills maintenance and competence are higher leverage than the usual focus on output checks.

Possible Architectural Responses

Here are some thoughts on patterns that could be materially better than the current default in many of the deployments I have seen.

Risk-tier deployments honestly. A lot of AI governance applies uniform review processes across tasks of radically different risk. Agentic architectures with thin oversight seem appropriate for low-consequence, reversible, high-volume tasks. Controlled architectures with genuine human authority (or lower risk old school deterministic logic) look more appropriate where consequences matter.

Slow the agent, do not speed up the human. If a human cannot plausibly review 200 outputs an hour, designing a system that produces 200 outputs an hour and then asking them to review it is probably not the right architectural choice (and going back to cybernetics goes against Ashby’s Law of requisite variety). Rate-limiting the AI rather than the reviewer could be considered though it sits uncomfortably with productivity and speeding up process narratives.

Budget for the J-curve. If the productivity case requires substantial organisational complementary investment (process redesign, skills development, data quality, governance maturity) the delivery plan probably needs to include those investments explicitly.

Budget for skills maintenance. If AI is automating the work that built a reviewer’s judgement, that reviewer capability is on a depreciation curve. Either invest deliberately in rotation back into unassisted work (chaos anti AI monkey maybe isn’t sounding so wild after all?!), or accept that oversight capability is decaying. Pilots have known this for decades. Knowledge workers may be about to rediscover it.

Design for obvious failure. Graceful, plausible failure is probably a confirmation-fatigue trap. AI systems that fail in ways that look obviously wrong rather than plausibly right may be more honest to design for, even though that conflicts with UX conventions optimised for polish (and LLMs nasty habit for sycophancy).

Flag recursive complacency as a known unknown. AI evaluating AI is useful but probably not a substitute for cognitive independence. I would be cautious about architectures that treat LLM-as-judge as solving oversight rather than augmenting it.

The chaos anti-AI monkey. I floated this in the original sleepwalking piece partly in jest, but the case for it gets stronger the more I think about it. Scheduled AI-off periods seem to surface hidden dependencies, maintain unassisted competence, and give reviewers empirical grounding for what “normal” looks like without the tooling. Something like a disaster recovery drill for cognitive infrastructure?

Does AI automation always make sense?!

Taken together, the three fallacies produce a question I do not hear asked nearly enough in AI governance conversations:

If the AI’s judgement is better than a human’s, the productivity gains are smaller than forecast, and a human reviewer cannot catch errors: what exactly is this AI deployment achieving? Is it worth the costs and risks?

Local gains (or massive volume tasks that would never have been possible for a human team) may still justify deployment. Some tasks are low-consequence enough that thin oversight is low risk. Some deployments are pilot investments whose real value is learning how to do the next one better. These are all defensible positions, and I would encourage architects to state them as the actual rationale rather than dressing them up as oven ready large-scale transformation(s).

What I find harder to defend is the tacit position many programmes (or vendor / consultant marketing narratives) seem to occupy: that substitution is clean, productivity gains are imminent, and oversight can be meaningful. The literatures on each of those questions do not obviously support all three being true at once. They may be true in particular cases (as I said earlier perhaps at large scales).

Bainbridge closed her 1983 paper with a line I have been thinking about:

“The difficulty remains that they are less effective when under time pressure… resolving them will require even greater technological ingenuity than does classic automation.”

The question, forty-three years on, is whether we apply it to the genuinely hard problem of designing in honest oversight, or whether we keep scaling systems whose governance is ceremonial and whose productivity case may turn out to be a vendor hype. I had been thinking that maybe we just need smarter feedback loops (leaning on cybernetics again) but after reviewing this literature I need to do a lot more thinking about how viable some adaptive / dynamic ecosystem architecture approaches might be.

Citations / Further reading

• Bainbridge, L. (1983). “Ironies of Automation”. Automatica, 19(6).
• Brynjolfsson, E., Rock, D. & Syverson, C. (2021). “The Productivity J-Curve: How Intangibles Complement General Purpose Technologies”. AEJ: Macroeconomics.
• Brynjolfsson, E., Li, D. & Raymond, L. (2023). “Generative AI at Work”. NBER Working Paper 31161.
• Dell’Acqua, F. et al. (2023). “Navigating the Jagged Technological Frontier”. Harvard Business School Working Paper 24-013.
• Daron Acemoglu, “The Simple Macroeconomics of AI,” NBER Working Paper 32487 (2024), https://doi.org/10.3386/w32487.
• Elish, M. C. (2019). “Moral Crumple Zones: Cautionary Tales in Human-Robot Interaction”. Engaging Science, Technology, and Society, 5.
• Cronk, O. (2025). “Are we sleepwalking into tomorrow’s AI challenges?”

Notes: This piece is a follow-up exploration of the Architect Tomorrow sleepwalking risks framework. Several claims here are hypotheses flagged deliberately as open (in particular the compounding dynamic across the three fallacies, the recursive complacency concern in LLM-as-judge patterns, and the load-bearing role of Relevance Degradation in the sleepwalking diagram). AI use disclosure - Claude Opus 4.7 used to help me find further relevant research and papers to the ones originally found and helped me elaborate and edit the piece. The whole piece has been through manual review and manual reference checking (would be a further irony otherwise!!!)

On 12 April, the Cloud Security Alliance, SANS Institute, OWASP, and [un]prompted jointly published a draft strategy briefing titled “The AI Vulnerability Storm: Building a Mythos-ready Security Program”. The contributing author list is impressive including: Bruce Schneier, Jen Easterly (former CISA Director), Chris Inglis (former National Cyber Director), Heather Adkins (CISO, Google), Phil Venables (former CISO, Google Cloud), Rob Joyce (former NSA Cybersecurity Director), and many more. This isn’t vendor marketing. It’s a cross-industry call to action, and the architectural implications are worth reflecting on…

The short version of what happened

Anthropic’s Claude Mythos (Preview) autonomously discovered thousands of zero-day vulnerabilities across every major operating system and browser, generating working exploits with a 72% success rate. It found a 27-year-old bug in OpenBSD. It chained multiple memory corruption vulnerabilities into single exploit paths without human guidance. Anthropic responded with Project Glasswing, giving 40 vendors early access so they can remediate.

Most of these capabilities predated Mythos, but Mythos has arguably made the threat harder to ignore. The briefing’s timeline shows this has been building since mid-2025: XBOW topping HackerOne’s leaderboard, Google Big Sleep finding 20 real-world zero-days, DARPA AIxCC finding 54 vulnerabilities in four hours, and state-sponsored groups running autonomous attack chains.

Why architects should care, not just CISOs

The attacker-defender asymmetry is now structural, not circumstantial. AI lowers the cost and skill floor for discovering and weaponising vulnerabilities faster than organisations can patch them. The paper is blunt: current patch cycles, response processes, and risk metrics were not built for this environment. For architects, this means the assumptions underpinning your non-functional requirements for security response times, patching windows, and incident frequency are likely wrong. If your architecture depends on a 30-day patch cycle as an acceptable risk window, that window is closing rapidly if hasn’t already closed.

Here’s the thing though: patching and IT hygiene have been neglected for a very long time, and this predates AI entirely. From my time working at Tanium, I saw how many large enterprises struggled with the basics: incomplete asset inventories, patching backlogs measured in months, and a persistent gap between what the security team thought was deployed and what was actually running. Mythos doesn’t create that problem; it weaponises it. AI can now finds vulnerabilities and exploit them, and compressed the timeline for adversaries to do the same. If your organisation hasn’t been disciplined about patching and asset management before now, the urgency just increased by an order of magnitude.

Agents are simultaneously the problem and the proposed solution. The paper’s priority actions include both “Require AI Agent Adoption” across all security functions and “Defend Your Agents” as critical priorities, to be started this week. The briefing explicitly states that without agents, most of its recommendations are untenable, but also that agents are “privileged, insecure by default, and not covered by existing security controls.” They recommend defining scope boundaries, blast-radius limits, escalation logic, and human override mechanisms before deploying agents in or adjacent to production.

This is the architectural challenge in a nutshell: agents need explicit lifecycle management. Spawn with scope, budget with token and time limits, monitor with real-time telemetry, and terminate when boundaries are reached. The CSA paper says you cannot treat agents as just another user or just another service. They are a new asset class requiring a new control framework.

The software supply chain just got significantly more dangerous. The paper calls out MCP servers, plugins, and agentic supply chains by name. One of its ten diagnostic questions asks whether you have “disciplined control repos, artifacts, and software, including for agentic supply chain such as MCP servers, plugins, and skills.” For enterprise architects managing integration landscapes, this is a material new dimension. Every MCP server, every tool definition, every retrieval pipeline is now part of your attack surface.

The architectural implications

The briefing recommends using AI agents offensively against your own code, in your CI/CD pipelines, and across security operations. That’s a significant endorsement of agentic patterns, but notice the framing: these are agents pointed inward, under controlled orchestration, with explicit scope and governance. It’s not a free-for-all.

The case for a governed orchestration approach, where a deterministic workflow engine selectively invokes AI rather than the other way round, becomes even more compelling in this context. If your vulnerability discovery agents, your code review agents, and your incident response agents are all operating through a governed orchestration layer with token budgets, audit trails, and kill switches, you are in a fundamentally different position than if you’ve handed AI coding tools the keys and hoped for the best.

This also maps directly to the sleepwalking risks we’ve been exploring. The “Asymmetrical Overload” risk, where AI-augmented actors overwhelm those without equivalent capabilities, is precisely what the CSA paper describes. But there’s a new sleepwalking risk emerging here: organisations that rush to deploy AI agents for defence without the governance structures to control them may be introducing as much risk as they’re mitigating. The paper itself acknowledges this, warning against waiting for industry governance frameworks. “Define your own now,” it says.

The human cost matters

Credit to the authors for addressing burnout directly. Security teams are caught in a vice: AI is simultaneously accelerating the volume of vulnerabilities they must respond to, the volume of code their organisations are shipping, and expanding the attack surface. The paper recommends requesting additional headcount and budget for reserve capacity. It also makes the observation that every security role is becoming an “AI builder” role.

This resonates with a point I’ve been making: the skills gap isn’t about learning to code. It’s about learning to work with AI as a collaborator. The organisations that will navigate this best are those that invest in their people alongside their tooling.

My take

The paper is right that Mythos represents a step-change headline grabbing situation, but it’s also candid that most of these capabilities predated it. Architects should be cautious about treating this as a singular event rather than a trend that’s been building for over a year.

The Y2K comparison in the conclusions is apt in one respect: it was a systemic threat the industry met through coordinated effort. But Y2K had a fixed deadline. This doesn’t. The paper’s own framing of “the first of many waves” is more honest and more useful for strategic planning.

There’s also a pattern here that architects should recognise and challenge. The paper’s core recommendation is essentially: the release of a powerful new AI capability has created significant new risks, and the primary mitigation is to deploy more AI. More agents, more AI-driven scanning, more AI-augmented response. There is a circularity to this that deserves honest examination. We are being told that the antidote to AI-generated risk is more AI, with all the second-order consequences that entails: greater complexity, expanded attack surfaces from the defensive agents themselves, increased energy consumption, deeper dependency on a small number of frontier model providers, and a skills landscape that shifts faster than most organisations can adapt. None of this means the recommendations are wrong. They may well be necessary. But architects should go in with eyes open about the compounding effects rather than treating AI-for-defence as a clean solution to AI-as-threat.

Finally, the recommendation to deploy AI agents across all security functions, whilst simultaneously warning that defensive AI technologies are “lagging behind offensive ones,” creates an interesting challenge. You’re being told to adopt tools that aren’t fully mature, at speed, under pressure. That’s precisely the environment where architectural discipline, governed orchestration, explicit boundaries, and human oversight, matters most.

The full draft briefing is available from the Cloud Security Alliance. I’d encourage you to read it, share it with your security teams, and use it to start a conversation with your CISO about what your architecture needs to absorb.

What’s your organisation doing in response? Are you seeing these pressures already? I’d love to hear from the community.

Oliver Cronk is Founder, Fractional CTO and Chief Architect at Cronk Advisory, and founder and host of Architect Tomorrow.

AI Use Disclosure: Claude Opus 4.6 assisted with the drafting this edition of the newsletter.

As a result of that we are actually releasing 2 episodes today to make up for the lack of content this year!

 The first is the special edition covering: Green IO London 2025 Conference Interviews and Highlights

The second is Architecture Governance with Selena, Darryl and Grant: Governance Is Your Innovation Engine (If You’re Doing It Right) with Grant Ecker

 As always these episode are making their way on to audio services now as well.

Green IO London 2025:

So another special edition of Architect Tomorrow for 2025 Green IO conference in London. Whilst we couldn’t capture everything, this episode gives you a taster of what was shared on the day - with interviews with some of the speakers and of course, Gael Duez!

Massive thanks to Gael and the Green IO speakers for working with me on this episode  - I think this might be the record for most guests on a single recording of AT!

Gaël DUEZ

Amael Parreaux-Ey @Resilio

Magali Saúl @ Sopht

Anne Chow

Mark Buss @ Ovo and Charlie Beharrell @ heata

Dryden Williams ams CEO CarbonRunner (at the time of recording - see note below)

Hannah Smith Director of Ops Green Web Foundation

Oliver Winks @ Root and Branch

Ian Brooks @ UWE

 If you’ve already watch the version of this that went out on the GreenIO podcast note that this has a bonus extra clip from me on the networking impacts panel I ran (this is at 23 mins in if you want to skip to just that as you’ve already watched the Green IO version).

Embedded LinkedIn content: view it on the original article

RIP CarbonRunner

Something that should be noted about this episode is that it was recorded before the sad news about CarbonRunner ceasing operations.

Was CarbonRunner a business or a design pattern?

I’ve had lots of ideas that I thought could be businesses during my career. Aside from fear of failure the main reason I’ve not executed on them - they didn’t stand commercial tests (they might be “good” ideas technically but what is the product market fit and is there demand? I do wonder (but of course don’t know as I’ve not spoken to Dryden since this episode was recorded in 2025) whether this was the case for CarbonRunner as well. It was a wonderful idea but was there enough of a commercially defensible moat around it? I wonder whether in time Dryden might be open to licensing or open sourcing the IP behind the CarbonRunner algorithm. There is also of course the Carbon aware vs Grid aware piece of this that needs consideration (but I know Dryden was focused on consequentiality rather than just attributional reporting impact when he presented at GreenIO).

Governance Is Your Innovation Engine (If You’re Doing It Right) with Grant Ecker of Chief Architect Network

Embedded LinkedIn content: view it on the original article

This is rather timely given Architect Tomorrow will be at Chief Architect Network London with Grant Ecker). This is the second episode with the new semi-regular line up!

Good governance unlocks your innovation engine. So why are we still getting it so wrong?

It was a throwaway line in a previous episode. Selena Evans observed that bad governance certainly slows things down, but good governance unlocks your innovation engine. Enough people responded to it that I wanted to dig in properly. So I brought Selena and Darryl Carr back, and added Grant Ecker, founder of the Chief Architect Network, to the conversation.

What followed was one of the most practically useful conversations we have had on the podcast. Here is what stuck.

Governance has a reputation it does not fully deserve

The “department of no” framing is real, but it describes governance done badly, not governance done well. The bad version abounds because it is easier to build:

“Governance can tend to be so bureaucratic and rules-based and oriented around committees and cross-functional alignment that structures can make things near impossible.”

The better version looks completely different. It is a learning function. A memory function. It answers the question: what information do we need, in what context, to make the right decisions and achieve our objectives? When governance is built around that question rather than around compliance checklists, something shifts. It stops feeling like friction and starts feeling like alignment.

Someone’s no is always someone else’s yes

When governance says no to a proposal, there is always a reason, and that reason is usually that it is saying yes to something else. Consolidating cloud infrastructure. Protecting operational cost. Maintaining consistency across a portfolio.

“The governance should be set up to help the company achieve its goals aligned to that strategy, and to challenge when those two things are out of alignment.”

The governance that earns a bad reputation is the governance that forgets this. When an ARB says no without articulating what it is saying yes to, it becomes an obstacle. When the organisation moves on from a strategy but the governance has not updated its rubric, it becomes the enemy. Grant’s practical tip: pay attention to your escalations. They are telling you your governance is out of step with what the organisation actually needs.

AI: same questions, harder game

Does AI fundamentally change governance, or is it just another technology to scrutinise carefully? Darryl’s answer was measured: yes and no.

“We still need to understand what the purpose is of introducing a new capability into the organisation. We need to understand how it delivers value and the consequences of the decisions we’re making around it.”

The fundamentals hold. The subject matter changes. What is different with AI, particularly with agentic AI, is that you are no longer governing a system that follows discrete rules. You are governing a system that might take unpredicted action. Grant describes needing to “think in 3D chess.” You have to teach these things how to learn, how to think, how to act, and then govern those things. That is a genuinely new dimension.

Selena’s instinct here: resist anthropomorphising. These systems have a specific kind of intelligence built on specific mathematical parameters. You can put certain risk mitigations around them. Treating them as human-like decision-makers is where organisations get into trouble.

Her preferred reframe: stop saying “human in the loop.” Say “AI in the loop” instead, and keep it that way. It is a small linguistic shift with significant implications for how you design governance around these systems.

Feedback loops: the thing we consistently get wrong

Running through the whole conversation was a theme about feedback. Not collecting feedback, actually closing loops and acting on what you learn. As Selena puts it:

“We need to be sensing more holistically and more continuously around feedback loops, both in terms of what is happening, how things are changing, the external environment, the internal environment. Those feedback loops degrade over time, so they require upkeep.”

This is where traditional compliance-oriented governance tends to fail. It was built for a stable environment. It does not have good mechanisms for continuous sensing and adjustment. AI makes this more urgent, not less.

The uncomfortable question

If governance exists to enable good decisions, to bring the right information to the right people at the right moment, then most architecture governance is not really doing that. It is doing something that resembles it from the outside: committees, review boards, approval gates. But the underlying function is missing.

The organisations that get this right treat governance as infrastructure for decision-making, not as a control mechanism. They build feedback loops. They align their rubrics to current strategy, not last year’s. They differentiate between the parts of the architecture that need standardisation and the parts that need room to experiment. They make sure that when governance says no, everyone in the room understands what it is saying yes to.

The ones that get it wrong are still running the immune system of 2015 on the challenges of 2026.

So: does your governance framework know what it is trying to enable? And when did you last check?

Listen to the full episodes on the Architect Tomorrow podcast. Find Grant Ecker and the Chief Architect Network at chiefarchitectnetwork.com. The previous episode with Selena and Darryl, Architecting Human-Centred AI, is available now.

Massive thanks to them for doing this - they have already committed to appearing in more episodes which I am super excited about - look out for those in 2026!

Feel free to watch the episode below, check it out on your favourite audio podcast service and/or read the rest of this article for a summary of some of the key points…

Embedded LinkedIn content: view it on the original article

The $2 Trillion Question: Where’s the Business Case?

The conversation opened with a stark reality check about the economics underpinning current AI investments. Darryl didn’t mince words about recent industry layoffs:

“What gets lost in that story is that there’s 11,000 people who don’t have a job tomorrow, and what does that mean, and why doesn’t that register in the conversations that go alongside it. I think that’s a significant problem in the way that we’re currently having these conversations.”

Oliver brought data to the discussion, highlighting a staggering disconnect: “These things need to be generating about $2 trillion worth of revenue by 2030. It’s just staggering numbers… the whole of cloud computing is worth something like $200 billion, or not even that… So it’s like an order of magnitude more.” [Ed: Post recording note- seems Oliver’s numbers are a little old with Cloud now around $900 billion but the point still stands that AI will need to be double the current entire cloud market]

The mathematics simply doesn’t add up. Current proven use cases—primarily code generation tools like GitHub Copilot with 29% acceptance rates—generate only billions in revenue. Yet the industry needs trillions to justify current investments.

The LLM Reality Check: Pattern Matching vs. Problem Solving

Selena brought crucial perspective on the limitations of large language models, challenging the prevailing narrative:

“Almost all of the narrative that we see in the public is based on large language models… And large language models are very limited in what they can do. They’re so fun, and I love using them, and the pattern matching, it’s incredible. I think that there’s tremendous value in decision support, with them, but that isn’t how people are using them. It’s talking so much about automation, replacing jobs, and I don’t think that we’re at a place where we even have the capabilities to do that.”

She drew on historical context to make her point more forcefully:

“Look at our older architecture work where we’re relying on RPA and regular algorithmic work in business, and we didn’t… we weren’t able to get a lot of traction on that anyway, because breaking down complex tasks into logic that can be executed over time is really challenging. And LLMs break down in complexity over time, and they drift, and all of these things that need to be managed on the tail end that isn’t getting any in the hype.”

The conclusion? The financials don’t make sense, the narrative doesn’t make sense, and as Evans put it: “We’re all in this new space where we’re buying into a lot of the narratives, because we don’t have the language to really… and proof points, to really, really talk about it smartly.”

The Enterprise Reality: Legacy Systems and Complexity

Darryl brought the discussion firmly back to ground with a dose of enterprise reality:

“I used to work in the banking industry, and I cannot for the love of me, see how anybody would let loose an AI in an environment running 40-year-old COBOL systems on mainframes. That’s not going to happen, so the built environment and the business cases required to support that activity don’t stack up.”

Whilst Oliver did see value in using these tools to help understand legacy systems he refined this perspective with a nuanced view of where AI might actually provide easier value:

“The brownfield use cases, where you’ve got the enterprise mess of people, process, existing technology… I totally think that space is really, really hard for this to crack. I think creating some marketing, sort of, front-end thing, you know, very straightforward… it’s gonna knock it out of the park, because it’s just rinsing and repeating a load of code that it’s been trained on. But this messy reality of most enterprises, I think, is where these things are… the reality’s just hard to unpick.”

The Forgotten Foundation: Data and Infrastructure

Selena highlighted a critical challenge that doesn’t make headlines but determines success or failure:

“I can make a huge case that the infrastructure in so many companies is inefficient to be able to support AI. Like, just data fragmentation on its own is a huge project, but it is not anything that they get splashy value from the street and the quarterly returns… finding executives to really dig in and then stick with the really long and hard work as priorities change, and as people change jobs, and as power dynamics shift, it’s really a challenge to keep these exceptionally important foundational projects going and aligned.”

Darryl reminded everyone of lessons that should have been learned:

“Every time I’d have a conversation with somebody around RPA, I would say, you’ve got two rules to follow. Don’t automate a process when you should be integrating systems. And don’t automate a bad process. It’ll just let you do something wrong quicker. So, have we fixed those problems? No, we haven’t. We’re just layering more technology over the top of it.”

Oliver Cronk summed it up: “AI amplifies what you already have. And if it’s bad, well, it’s gonna get worse.”

FOMO and Timing: The Kodak Moment Trap

The conversation tackled a genuine concern: how do organisations avoid being left behind whilst maintaining sanity?

Oliver acknowledged the tension: “Organisations that didn’t time their transformation to e-commerce right… there’s a whole load of examples… Like, Amazon started by eating up the book market, but then went on to sell a whole bunch of other things. There’s a whole load of industries that we could kind of go and look at that’s had their Kodak moment, because they didn’t digitize.”

But he also urged caution: “Timing, for me, is, like, one of the most important things… does it make sense for you right now, or does it make sense to sort of pause, reflect, see what’s going on in the market, when are things mature?”

Ecosystem Architecture: Looking Beyond Your Walls

Selena offered a pragmatic framework for navigating uncertainty:

“I think that the immediacy is important, but then the slow deliberation, the hard work of building the models, the hard work of figuring out what it means for your industry, looking at second and third order concepts, and doing some serious futuring work around the ecosystem of your business is important for everybody.”

She continued: “Look at the entire ecosystem that you’re in and start to leave options open to yourself about how you can move through this and make sure that you’re structured to even be able to hear the signals… the truthy signals from the market, to be able to time them correctly. And that’s what architecture is so amazing, right? This long-term, short-term balance and scaffolding to move as you want to, that’s the beauty of it.”

Oliver strongly endorsed this shift: “Ecosystem architecture has come up time and time again on this podcast, and I think it’s this… I genuinely think it’s where the enterprise architect role needs to shift to. It’s like, stop just looking just within your interior boundaries, and think about what the market looks like, what the competitors look like, what your potential partners look like.”

The Human Work Technology Can’t Replace

Selena delivered perhaps the most important reminder of the entire conversation:

“Deliberate analysis and really hard, multidisciplinary work… coming up with shared meaning amongst different professionals that have entirely different worldviews. We, like, we suck at that in organizations. It is, like, a really, really challenging thing to get people to even, let alone open enough time in their schedules to be able to do that, but to get people to be that open-minded and not fall into politics traps, to be able to have those hard conversations… There’s no technology that can give… that can substitute that hard work. None.”

Flipping the Script: What if We Started with Society?

As the conversation shifted to longer-term futures, Selena challenged everyone to think differently:

“I would love if we flipped the script. What do we need to do as society to better our lot in life? What do we need to do to address the climate horizons that we are facing? What do we need to do to ensure flourishing of human and other species and biodiversity on our planet? And let’s commercialize those things. Let’s use those frames to align incentives and align capital in ways that we haven’t before.”

She warned about the consequences of not changing course:

“What I think is very likely to happen, LLM bubble bursts, we realize that we’re not going to be able to solve all the world’s challenges with these LLMs. I kind of hope that it happens fast enough that the public gets really pissed off about the consequences of it, so that with this next wave of really cool technology, gets the proper governance systems built around it.”

Her critique was pointed: “If we’re teaching computers like babies, let’s teach them that they have to do X, Y, and Z, and teach them that these are the things that we, as a society, have decided are non-negotiable. Because there are very smart people who are very worried about what these technologies will be able to do, especially if we build them in this old, extractive capitalist logic, which looks at humans as a resource versus the people to be served.”

Creating Something Better: Willpower and Principles

Darryl offered both realism and hope: “There’s absolutely a set of foundational principles that we could apply, and build these types of organisations and this type of society that we’re talking about. At the moment, the balance of power is not in that favor… But absolutely, we created this mess. It didn’t exist a little bit more than a century ago… We’ve created a problem that we could otherwise have not created. So we have the ability to create something else. But we have to have the willpower to do that.”

He also warned about how modern life is reshaping our thinking: “If you look at the work of people like Ian McGilchrist, who understands very well how the different hemispheres of the brain work, and how we’re feeding one side of the brain and neglecting the other. And it’s the feedback loop that is creating the environment that we find ourselves in now.”

The Political Reality: Polarisation by Design

Selena didn’t shy away from addressing the elephant in the room:

“The polarization is by design. That is a piece of a playbook… people on different sides of the spectrum are getting two entirely different narratives about what is going on, so we don’t have a shared reality.”

But she also saw grounds for optimism: “The power has to rest with the people and with what we demand of our governments… The idea that states can create environments that serve their communities here in the U.S. is gaining a ton of traction, and there’s a lot of state and mayor collaborations that are going on that give a lot of positive signals. So I think it can be done, but I think it’s really hard work, and I think it takes a civic duty on behalf of all the individuals involved.”

Architects as Policy Makers

This led to a crucial reframing of the architect’s role by Selena:

“Architecture provides a policy framework. Like, to have an architecture-led policy framework that looks at the purpose of organizations and institutions… the whole idea that the purpose of a system is what it does, is something that you can tie to from a policy perspective. So yes, architects, lean in. Like, let’s create civic architectures that can help society thrive.”

Darryl added his perspective on why architects matter:

“One of the reasons that architects are useful in organizations is because we provide decision support. So we are able to see holistic views of the organization that allows better decisions to be made because more awareness is brought into the conversation. And one of the things that we use in that process are principles… we need to define better principles, and we need to be able to have the conversations where those principles are applied in the decisions being made.”

Governance: The Misunderstood Enabler

Selena took on the widespread misconception that governance slows things down:

“This idea that regulation and guardrails and governance just slows things down so much—that is not based on reality. Bad governance certainly slows things down… in the way that organizations practice it very defensively and with a frame of risk aversion at all costs. Like, there are different ways to mitigate risk that are not so bureaucratic and restrictive, and a lot of that happens at the design phase.”

She explained how governance should work: “If we look at risk and opportunity at the same time, and we build conscientiously and ethically from the up, like, it doesn’t become restrictive, it becomes what Daryl says, is a design principle that you operate by… I think governance unlocks your innovation engine. It unlocks your ability to align your enterprise. It helps you develop that shared meaning that’s going to allow change initiatives to actually be successful.”

Oliver added: “Give me the freedom of a tight brief… constraints drive innovation.” See his recent talk on balancing AI Innovation and Sustainability for more on that theme: https://blog.scottlogic.com/2025/11/12/balancing-ai-innovation-sustainability-hm-treasury-id25.html

Architectural Grafting: Incremental Transformation

Perhaps the most hopeful concept to emerge from the conversation was Selena’s notion of “architectural grafting”:

“I call it architectural grafting. That’s from an actual building architect who talks about using the base of old businesses and putting new ones on top, which I just love the concept. If we can do that over time, we’ll incrementally be in such a better place and build the foundations for what is possible in the future.”

This metaphor perfectly captures the pragmatic approach needed: respecting what exists whilst building towards something better.

Words of Hope in Anxious Times

As the conversation drew to a close, all three participants offered reassurance to those feeling overwhelmed by the current moment.

Darryl advocated for simplicity: “If you took anybody involved in the changes that are occurring now, whether it’s the typical person on the street, or the tech CEO, and you said, what do you want to be in 10 years or 20 years’ time? And the first thing they’ll say is they want to be around. And they want to be happy. And it’s just trying to align those needs… Take a breath. And have the conversations that describe what that’s going to be like, and then take action based on those discussions.”

He also reframed what technology should be for: “Bring on the technology. If it’s going to enable us to have more time to be a human being and share time with people that we enjoy spending time with, that’s fine. You know, take the jobs away. Give us the freedom to be something other than a machine in a factory, as we have been for the last 150 years or so.”

Selena found hope in the community of practitioners: “I think so many of us are feeling this existential angst, and that in and of itself opens the door to considering new possibilities. And so I find a lot of hope from so many dedicated people who are doing really great work… at least to me, it feels like like-minded folks are, like, across disciplines are starting to come together in new ways. And I think that that is going to unleash magical potential.”

She also identified a broader societal shift: “I think we already are sick of feeling so extracted from all the time, and so inundated by our consumerism, and that we’re craving more connection and community… I think that there’s a lot of good work trying to solve that. If we get there, that unleashes the potential for us to move forward in different and more thoughtful ways.”

Oliver reflected on the importance of perspective: “I think it’s easy at the moment to be sort of swept up by the narratives that we’re all consuming, and taking some time to sort of pause… I’ve deliberately been getting out of the LinkedIn bubble, because I felt like I was spending way too much time being influenced by LinkedIn.”

Selena had the final word with a call to action:

“You must keep up the hope… I think that we’re starting to kind of understand the broader patterns. I think that more and more people are starting to feel the effects of those larger destructive patterns, and so as this happens more and more, people’s minds are going to start changing, and we’re going to start demanding different things hopefully, of our governments, of our institutions, including business… And so then, then we architect tomorrow as minds expand.”

Key Takeaways for Architects & Strategists

1. Challenge the narrative - The economics of current AI investments don’t add up. Ask hard questions about ROI and business cases before committing resources.
2. Understand the limitations - LLMs are excellent at pattern matching and decision support, but they break down with complexity and drift over time. They’re not a replacement for human judgment in complex scenarios.
3. Fix the foundations first - Data quality, system integration, and organisational structure matter more than ever. AI amplifies what you have—for better or worse.
4. Think ecosystem, not just enterprise - Look beyond your organisational boundaries to understand market dynamics, partnerships, and how transformation will ripple through your industry.
5. Embrace governance as an enabler - Well-designed governance unlocks innovation rather than constraining it. Build it into the architecture from day one.
6. Practice architectural grafting - Build on existing foundations incrementally rather than attempting wholesale replacement. Society and organisations need time to digest change.
7. Keep humans central - No technology can replace the hard work of building shared meaning across disciplines and worldviews. This is where architects add irreplaceable value.
8. Define clear principles - Architects provide decision support by bringing holistic awareness to conversations. Establish and apply principles consistently.
9. Take a breath - The pressure to act immediately is real, but timing matters. Sometimes waiting for maturity is the right call.
10. Maintain hope through action - The situation is complex, but people across disciplines are coming together in new ways. Your work matters.

See also the previous newsletter on AI principles for more along these lines:

Embedded LinkedIn content: view it on the original article

The Path Forward

Oliver summed up the podcast’s mission: “Being hopeful, optimistic, and trying to build a better… is ultimately what this podcast is all about. Yes, there’s lots of complexity out there, there’s lots of challenges, but our role in the world is to try and nudge and push things in a more hopeful direction.”

Darryl reminded us of the choice we face: “Continue to be a human being.”

We hope we modelled something something rare and valuable: thoughtful, critical analysis combined with genuine hope for a human-centered future. We demonstrated that it’s possible to be skeptical of vendor narratives whilst remaining optimistic about technology’s potential to serve humanity.

The question isn’t whether AI will transform industries—it’s whether we’ll architect that transformation thoughtfully, inclusively, and sustainably, or whether we’ll sleepwalk into tomorrow’s challenges.

Embedded LinkedIn content: view it on the original article

As Selena Evans put it: “Then we architect tomorrow as minds expand.”

The choice is ours.

New: ArchitectTomorrow.com

In other news - Architect Tomorrow now has a proper website! https://architecttomorrow.com/

Proud that we are practising what we preach - it’s on Green Hosting and uses a minimal Architecture (Markdown generated Static HTML system via Cloudflare pages). In a bit of irony we created quite a bit of this using Claude and Google Antigravity AI tools, Front end development seems to work well with these tools if you specify what you are after reasonably precisely (it did require quite a bit of back and forth). If you spot issues please do let me know.

Closing notes:

This article is based on the Architect Tomorrow podcast episode and was created with the assistance of Claude.ai - using the transcript and then human review and editing.

Want to join the conversation? Connect with the Architect Tomorrow community and share your perspectives on human-centered architecture and technology transformation.

Mentioned Resources:

• “Reshuffle” book on ecosystem thinking
• “Whatever Next” by Lisa Woodall on transformation leadership
• Ian McGilchrist’s work on brain hemispheres and thinking
• Dark Matter Labs (civic innovation)
• Green IO conference on sustainable technology
• Oliver’s 2016 piece on algorithmic banking
• Various reports on AI ROI: Bain, Deutsche Bank, McKinsey
• Research on worldview models and visual learning in AI

This is the second part of my reflection on the Royal Society’s 75th anniversary of the Turing Test event (2 October 2025). Part 1 covered what happened at the event - the dismantling of AI hype, the evidence of scaling limitations, and the documented harms being ignored in the rush to deploy.

Your Role: The Long-Term Voice of Reason

You’re uniquely positioned (as an Architect / Tech Strategist) to influence business strategy, challenge vendor claims, and ensure system resilience. Whilst procurement chases the next shiny thing and leadership responds to market pressure, you’re thinking in architectural timescales - five, ten, fifteen years out.

That perspective matters now more than ever. The correction seems likely. Your job is ensuring your organisation protects genuine value when it arrives and that you’ve built systems on what actually works rather than vendor promises.

• Demand evidence over enthusiasm
• Design for reality over brochures
• Build systems that augment capability without degrading people’s capabilities
• Protect genuine value when the bubble deflates

The scaling wall appears real. The economics look challenging. The question is whether we’ll have an autumn (harvesting what works, dropping what doesn’t) or a winter (wholesale rejection losing genuine value).

Recap of AI Autumn

AI Autumn is my suggested next phase - where dead leaves (failed experiments) drop, we harvest what works (scale prototypes to production) , and we prepare for what’s next (the air coming out the bubble?). For more on this see the last article.

Six Principles for AI Strategy

1. Map Reality, Not Marketing

Current AI excels at content transformation with good data. That’s its strength. Systems tend to fail at causal reasoning, long-term planning, novel scenarios, and grounded truthfulness.

• Match use cases to actual capabilities, not vendor claims
• If you need robust planning or handling unexpected scenarios, current systems will disappoint you
• Stop procurement favouring “general purpose” platforms
• Demand proof in your specific context, not synthetic benchmarks

2. Protect Your Data Assets

Model collapse is documented. Systems trained on AI-generated content progressively deteriorate. The internet is contaminated. Clean, human-generated, domain-specific data becomes increasingly valuable.

• Don’t build strategies assuming infinite model improvement
• Plan for degradation
• Treat proprietary data sources as strategic assets
• Question where training data comes from

3. Invest in Specialisation

AlphaFold revolutionised protein folding. LLM’s haven’t replicated that success across complex highly regulated domains. Evidence suggests the best results come from highly specialised, modular systems, not jacks-of-all-trades. Consider the full range of AI techniques - symbolic reasoning, expert systems, statistical models, neural networks - rather than over-indexing on generative AI.

• Stop procurement favouring “one model to rule them all”
• Invest in domain-specific models where ROI is demonstrable
• Consider hybrid architectures combining different AI techniques (not just transformers)
• Match the AI approach to the problem (Gen AI for content transformation, expert systems for rules-based decisions, statistical models for prediction)
• Challenge vendors claiming general intelligence

4. Challenge AGI Rhetoric

Keeping AGI “always in the future, never now” lets companies evade transparency, ignore current harms, and justify massive investment. But do the economics actually work? Bain projects an $800B revenue shortfall by 2030.

• Reframe business cases around measurable outcomes
• Don’t accept AGI-adjacent promises
• Resist inevitability narratives
• Remember you have agency

5. Build Resilience

Consider second-order effects. Are you building dependency or capability? What happens when AI fails? Will junior staff ever learn to think critically?

• Design systems that can function when AI fails
• Consider “chaos anti-AI monkey” days to test resilience
• Question whether you’re augmenting capability or creating atrophy
• Map workforce skills against AI dependency

6. Make Externalities Visible

AI systems have environmental and social costs that rarely appear in business cases. Training large models consumes vast energy. Deployment at scale affects employment, widens inequality, and can enable surveillance. These aren’t abstract future concerns - they’re current costs being externalised.

• Include environmental impact in technology assessments
• Consider social impact alongside efficiency gains (job displacement, skill degradation, inequality)
• Question whose costs you’re externalising and whether that’s acceptable
• Ask who benefits and who bears the costs
• Make these trade-offs explicit in decision-making rather than pretending they don’t exist

The Workslop Problem

Gary Marcus referenced the term “workslop” for AI-generated content that looks acceptable but lacks substance. One result of this is confirmation fatigue where human review becomes meaningless theatre. We need to watch out for:

• Automation overload (systems designed for human use overwhelmed by agentic AI)
• Confirmation fatigue (reviewers rubber-stamping because everything looks plausible)
• Capability degradation (staff never developing expertise because AI does it)

What to Stop

Brute-force approaches:

• “Scaling is all you need” - but have GPT-5 and Llama-4 delivered on their hyped expectations?
• Mass deployment without evidence
• Procurement favouring general-purpose platforms

AGI theatre:

• Business cases built on AGI promises
• Inevitability narratives that remove agency
• Vendor claims about reasoning or intelligence without proof

Ignoring externalities:

• Business cases that ignore environmental costs
• Deployment decisions that externalise social harms
• Technology assessments that only count benefits, not costs

What to Start

Systematic evidence:

• Carefully managed testing in real-world like contexts where you’ll deploy
• Demand demonstrable ROI in your domain or experiment to validate claims
• Build evaluation frameworks that test actual use cases

Strategic investment:

• Domain-specific tools with proven outcomes
• Various types of AI matched to the problem (Gen AI for content transformation, expert systems for rules, statistical models for prediction)
• Protect data assets as model collapse progresses
• Multimodal integration only where it adds genuine value
• Don’t over-index on Gen AI whilst ignoring other proven techniques

Resilience planning:

• Maintain ability to function when AI fails
• Differentiate research from deployment
• Consider workforce capability alongside system capability
• Build in human expertise development, not just AI augmentation

Externalities accounting:

• Include environmental impact in technology decisions
• Make social costs visible in business cases
• Question who benefits and who bears the costs
• Consider whether you’re solving problems or shifting them elsewhere

What would you add to this? Keen to hear your thoughts.

There was an emphasis on a variety of AI models and techniques and a more naunced / considered definition of what intelligence actually is. They argued that whilst large language models may pass surface-level Turing tests, they remain fundamentally limited pattern-matchers. Another important question is whether current AI deployment represents what Professor Sir Nigel Shadbolt termed a “massive uncontrolled experiment” on society, with insufficient regard for safety, ethics, or human flourishing.

I’ve pulled this piece together from my notes on the event and some of things I’ve written about on this topic on the Scott Logic Blog and this Architect Tomorrow newsletter. BTW a part 2 more on the so what for Architects? Is now available.

The Scaling “Law” is actually hitting a Scaling Wall

Recently the mantra has been: bigger models, more compute, more data = better results. Rich Sutton’s 2019 essay “The Bitter Lesson” argued that raw scale always wins over clever engineering. The tech industry bet trillions on this assumption.

Dr Gary Marcus widely dismissed as a sceptic, has been vindicated somewhat. When he argued in 2022 that scaling laws weren’t universal, Sam Altman called him a “mediocre deep learning sceptic.” Even Rich Sutton himself publicly acknowledged the limitations recently, tweeting: “You were never alone, Gary… I salute you for this good service.”

The timeline of collapse tells its own story. Satya Nadella in November 2024: “There is a lot of debate… have we hit the wall with scaling laws? These are not physical laws, just empirical observations.” Llama-4 in April 2025 “landed with a thud.” GPT-5 arrived to mixed reviews - certainly not the level of AGI progess that had been hyped.

The economics tell a similar story. Bloomberg and Bain analysis suggests AI companies will need $2 trillion in combined annual revenue by 2030 to fund computing power, but are likely to fall $800 billion short (btw the entire public cloud market is round $200B).

Embedded LinkedIn content: view it on the original article

Personally I think these models do have value - and some of the lack of ROI reports that featured at the event were a little bit cherry picked. However the value at the moment is more about augmentation - with hard to measure value. I think the jury is still out on the the drive for more autonomous agentic approaches.

Pattern-Matching Isn’t Thinking

Marcus demonstrated what systematic testing reveals: LLMs regularly fail in ways that expose fundamental lack of understanding. Examples shown at the event included GPT-5 placing random labels on elephant images with bizarre misspellings, dangerous electrical wiring advice that would cause short circuits, and complete spatial confusion when asked to show five clocks at specific times (rather than the most common 10 past 10 image).

These aren’t edge cases. They’re predictable failures revealing that these systems are curve-fitters finding correlations without understanding causation. As Alan Kay demonstrated, a 0.99 correlation between divorce rates in Maine and margarine consumption doesn’t mean one causes the other, yet AI systems trained to optimise for pattern-matching cannot distinguish coincidence from causation.

Dr Stevan Harnad framed it clearly: LLMs are “not understanding anything, it’s false.” The symbol grounding problem remains unsolved. These systems manipulate tokens without comprehending what those tokens represent in the physical world.

Known Harms, Ignored Warnings

Contrary to rhetoric about “harms we can’t anticipate,” the problems are well-documented:

• 2016: Bias issues extensively called out
• 10 years ago: Deskilling raised as major risk
• 1966: ELIZA effect (overattributing intelligence to simple systems) identified

Dr Kaitlyn Regehr’s research reveals a troubling generational divide. Older users employ AI to support existing expertise. Younger users rely on it instead of developing expertise. A 15-year-old girl now uses ChatGPT to text her friends because “I think it does a better job than I can.” She’s losing confidence in her own communication abilities.

More seriously, there have been documented cases of AI companion-induced suicides among teenagers, where systems provided detailed instructions for self-harm. Dr Abeba Birhane highlighted that these systems also amplify existing biases. Machine learning inherently encodes hierarchies where privileged groups consistently rank higher, with applications in hiring, medical diagnosis, and law enforcement exaggerating these patterns.

What Actually Works: Specialisation Over Generalisation

The counterexample to LLM hype is DeepMind’s AlphaFold 3. Purpose-built for protein structure prediction, combining multiple specialised approaches, it has genuinely advanced scientific understanding. Marcus described it as “exquisite” engineering.

The lesson: “Some of the best results in AI have come from highly-specialised, highly modular systems.” Not jacks-of-all-trades attempting everything adequately, but focused tools excelling at specific problems.

Slide from Nigel Shadbolt about varieties of AI

This aligns with cognitive science. Jerry Fodor’s The Modularity of Mind argues human intelligence isn’t a single general-purpose system but many specialised circuits working together. Professor Shannon Vallor suggested we should decompose “intelligence” into specific capabilities rather than treating it as a unified concept, likening it to phlogiston (a historical scientific concept we now understand was misconceived).

The neurosymbolic approach (that I recently wrote about combines pattern recognition (what current AI does well) with symbolic reasoning (what it does poorly). Statistical learning excels at perception but struggles with logic. Rule-based systems excel at reasoning but struggle with adaptation. Combining both offers a more promising path than simply scaling transformers.

Embedded LinkedIn content: view it on the original article

The Uncontrolled Experiment

No other field would permit such unchecked deployment. Aviation, pharmaceuticals, and transport all require rigorous testing before public rollout. AI systems face no such constraints.

Dame Wendy Hall opened the event with a stark observation: “Don’t worry about AI, worry about low levels of human intelligence.” We’re remarkably easy to fool. What Turing arguably got wrong was overestimating human intelligence. The test measures our gullibility more than machine capability.

Embedded LinkedIn content: view it on the original article

Alan Kay expanded this point. Humans don’t just get fooled, we pay to be fooled. Theatre, television, advertising all demonstrate we inhabit a “waking hallucinatory dream.” This isn’t a bug; it’s how we function. AI systems exploit this vulnerability at unprecedented scale.

Kay’s most chilling point (but possibly a little elitist?): “The most frightening thing I can think of is trillions of ordinary human-level intelligences roaming the internet at will.” The threat isn’t superhuman AI but scaled human stupidity, amplified by systems that inherit our biases and lack our contextual understanding.

Kay invoked the engineering principle: “The bridge must not collapse. The plane must not crash. The software must not harm or fail.” AI development needs duty of care. A Hippocratic oath: “Do no harm.”

Resisting Inevitability

There was quite a lot of talk on the inevitability narrative. “AGI is coming whether we like it or not” serves corporate interests, not public good. It positions citizens as passengers rather than agents.

This is rhetorical strategy, not technical reality. Vallor emphasised: “It’s vital we resist the idea that we are passengers… AI is behind the wheel and we’re just hoping it takes us somewhere nice.” The lack of control is political and economic reality, not technological destiny.

Governments face enormous lobbying pressure. A $100 million political action committee backed by a16z and OpenAI aims to shape regulation in industry’s favour. Regulatory capture is real. But humans have resisted bad ideas before. We can do it again.

The question isn’t whether there will be a market correction (most investors recognises valuations are unsustainable) but how much damage occurs first. Social media offers a cautionary precedent: we normalised harmful technology because it was convenient and profitable, only addressing problems after significant societal damage.

What Needs to Happen

The path forward requires several shifts:

Safety must be paramount. Kay argued “safety should be the theme of the 21st century.” This means rigorous testing before deployment, liability frameworks that actually constrain behaviour, and investment in safety research across academia and industry.

Demand evidence. When universities, companies, or governments claim AI benefits, ask for data. Challenge the fear of missing out with requests for systematic evidence. Make deployment conditional on demonstrated value, not speculative promises.

Empower diverse voices. Dame Wendy Hall noted AI discussion is dominated by “tech bros” and politicians with limited diversity. Mothers, daughters, different ethnicities and cultures must be part of this debate because it affects everyone globally.

Protect creative work. The mass appropriation of content to train models represents an unprecedented transfer of value from creators to corporations. This requires urgent attention.

Resist “AGI” rhetoric. By keeping AGI “always in the future, never now,” companies ignore current harms, evade transparency demands, and sidestep responsibility. Draw a line under this framing.

Focus on augmentation, not replacement. The better question is how we augment human intelligence without making ourselves stupider. We made these tools. The challenge is thinking carefully about how they shape us.

My Reflection - the need for AI Autumn to avoid AI Winter:

Personally I see massive potential in a range of types of AI - but it’s important that we consider the value and costs of these when implementing. As per my recent post on the need for AI autumn. AI has value but there are economic, social, environmental and technical risks from the current approach - that could lead to an AI winter.

Embedded LinkedIn content: view it on the original article

• Dead leaves - being approaches or experiments that don’t show ROI or are brute forced and unsustainable
• Harvest the approaches that do work and encourage people to use a range of models and techniques that best fit the problem being solved
• Crucially do these things in a way that benefits people and enterprises not just driving centralisation of profits and progress in a single direction
• Avoid a knee jerk reaction when the air comes out of the bubble that drives a once-bitten-twice-shy reaction to markets and businesses when it comes to broader AI.

Conclusion

Seventy-five years after Turing asked “Can machines think?”, this gathering suggested it’s the wrong question. Better questions: What exactly does this machine do? Does it work reliably in real-world contexts? Who benefits and who bears costs? Does it add meaningful value to human lives?

The Turing Test has been passed and proved largely irrelevant. Current AI systems are powerful pattern-matchers, not thinkers. The pursuit of AGI may be chasing a mirage whilst ignoring present harms.

Most importantly: we are not passengers. Despite rhetoric of inevitability, despite trillion-dollar bets, despite regulatory capture, we have agency. We can demand better. We can resist. We can build the future we actually want rather than the one being imposed.

As Turing himself said: “We can only see a short distance ahead, but we can see plenty there that needs to be done.” Time to do it.

Here is part 2! Which is more of a so what for Enterprise and Technology Architects. If you have thoughts or feedback do leave or comment for the #ArchitectTomorrow community to discuss or send me a DM.

Jenny’s journey from studying animal behaviour to leading technology strategy at one of Europe’s largest DIY retailers offers a unique perspective on how biological thinking can inform better architectural decisions. In this wide-ranging conversation, she challenges conventional views on career paths, AI adoption, and the role of enterprise architects.

Embedded LinkedIn content: view it on the original article

Breaking the Technical Mould

“I am very, very clearly not a technologist. I would like to make that very clear. I have not done the engineering background route into EA. I’ve come from the other side of the spectrum—much more business focused.”

Jenny’s unconventional path from business analysis to chief architect highlights a crucial point about diversity in architecture teams. Rather than seeing non-technical backgrounds as limitations, she argues they bring essential complementary skills.

“I don’t want a single type of architect in the team. It doesn’t help anyone. It doesn’t challenge you as an individual. You’re all thinking in the same way. So that diversity is really key to me.”

Her advice for overcoming imposter syndrome? Focus on transferable skills rather than perceived gaps:

“The barrier to entry and the imposter syndrome is ‘I can do this. I can learn the stuff you need me to learn. Here’s the bucket of things I’ve got already that will magnify me in this role and will add value straight away.’“

The Double-Edged Sword of AI

Jenny is clear about AI’s environmental impact, describing it as “destroying ecosystems [through significant] land usage, water usage, power usage.” Yet she sees critical applications that justify the cost.

“Unfortunately, people like Google are going to be putting this into your search results. So you don’t have a say of whether it’s used or not. GenAI is there. It’s part of the world.”

The key distinction she makes is between generative AI tools (like ChatGPT) and more targeted narrow AI/ML applications:

“As you shift through the spectrum of complexity of AI, you start to talk about neural networks, machine learning, then you start to go, ‘Okay, this is the good stuff. This is the stuff that could solve some of the problems we’re seeing with the climate.’“

On AI augmentation versus replacement, Jenny emphasises maintaining authentic voice:

“I think the risk comes when you don’t have that confidence and this goes into the imposter syndrome… everyone almost becomes the same cookie cutter sort of mould.”

From Gatekeeping to Enabling

One of her strongest challenges is to traditional enterprise architecture approaches:

“Traditional architecture and traditional enterprise architecture is very much the ‘thou shalt not pass through this gateway unless I tell you it’s the right way to go.’ And I don’t think that’s ever been the right way to go.”

Instead, she advocates for architects as strategic enablers:

“My team aren’t there to make the decisions. They are there to help people make decisions because the ultimate accountability doesn’t sit with them. They don’t own the product or the platform, but they are there to help them make the best decisions.”

This shift requires systems thinking that considers broader impacts:

“At a macro level, the EAs look across everything and they go, ‘Well, if you make this decision, it will cause this to happen over there.’ And that’s where your systems thinking comes in.”

Evolutionary Thinking in Technology

Jenny’s zoology background provides a powerful framework for understanding technology ecosystems:

“Zoology is all about how animals interact with the environment around them. So it’s saying how does that animal, how’s that animal evolved to take advantage of the environment it’s currently within?”

This biological lens helps organisations navigate rapid change:

“The people that adapt and change will be the most successful. So it’s which businesses and what can you do within your business that you operate in to make the most out of what’s happening in front of you.”

Rather than fixed target states, she advocates for directional strategies:

“This isn’t a target state. This isn’t where the boxes must be in the future. This is the indicator of the direction of travel… we should probably head north because that’s our best opportunity to meet the strategic goals of Kingfisher.”

Practical Sustainability with Tech Carbon Standard

Jenny explained why Tech Carbon Standard resonated with Kingfisher’s approach to environmental responsibility:

“The tech carbon standard gave me that really clear upstream, operational, downstream view and all of the reporting requirements across that… I can clearly articulate this to people.”

Embedded LinkedIn content: view it on the original article

The framework provided actionable insights rather than abstract measurements: “What it gave me is that view on a page of saying here’s my footprint across this piece and here’s the levers I can pull on in each area to make a meaningful change.”

Key Takeaways

Embrace diverse backgrounds: technical depth isn’t the only path to architectural success (particularly Enterprise Architecture). Business analysis, scientific thinking, and other disciplines bring valuable perspectives.

AI requires nuance: environmental costs are real, but targeted applications in sustainability and decision support can justify the investment when applied thoughtfully.

Enable, don’t control: move from architectural gatekeeping to strategic enablement, helping stakeholders make informed decisions rather than imposing solutions.

Think evolutionarily: like biological systems, technology ecosystems must adapt to survive. Focus on fitness functions and adaptability over rigid target states.

Make sustainability actionable: Abstract carbon reporting becomes valuable when it clearly shows what levers you can actually pull to make meaningful changes.

Note this article was created from quotes of the episode transcript with the assistance of Claude and then human editing.

If you have an Architecture, Innovation and/or sustainability story you’d like to tell feel free to get in touch with Oliver Cronk

After two years of “throw LLMs at everything,” I’m noticing a shift. Many are quietly admitting that pure stochastic approaches have fundamental limitations when you need reliability, auditability, and real-world grounding. Which is interesting as it’s something we’ve been looking at for a while in a smaller way through the InferGPT -> InferESG journey.

Quick Primer: What’s NeuroSymbolic AI?

Think of it as combining the best of both worlds:

🧠 Pure Neural (current LLMs): Pattern recognition, creativity, but “black box” reasoning

⚙️ Pure Symbolic (traditional systems): Logical rules, explainable, but brittle and narrow

🔗 NeuroSymbolic: Neural networks for perception + symbolic reasoning for logic = grounded, explainable AI

What I’m Observing

The pattern is remarkably consistent:

• “Our AI hallucinates when we need facts”
• “We can’t audit statistical relationships”
• “Vector embeddings alone don’t give us compliance trails”
• “How do we validate outputs against our actual business rules?”

A key question for me:

Why are we training AIs purely in vector space (statistical relationships) when we could be building conceptual world view models alongside them?

What if we took the same training data - Common Crawl, enterprise documents, whatever - and systematically processed it to extract structured knowledge graphs? Concrete entities, verifiable relationships, formal properties?

Yes, it would require human input. Yes, there’d be debates over “ground truth.” But isn’t that exactly what robust AI should have - rigorous, auditable foundations?

Is This NeuroSymbolic’s Moment?

I’m wondering if we’re finally reaching the point where the limitations of purely stochastic approaches are forcing a rethink:

❓ Accuracy: Can we cross-validate vector outputs against structured knowledge?

❓ Costs: Could symbolic reasoning reduce expensive model calls?

❓ Governance: Do we need explainable reasoning chains for compliance?

❓ Sustainability: Are “brute force” approaches hitting natural limits?

What I’m Seeing in Practice

Some organisations (including my own via the R&D and client work we’ve been doing) are already quietly building hybrid architectures:

• Knowledge graphs validating LLM outputs
• Structured queries enriching vector search
• Rule engines checking AI decisions against policies
• Human experts curating verified knowledge bases

Are these the early signals of a broader shift toward NeuroSymbolic AI?

The Bigger Question

Are we witnessing the end of the “statistical everything” era in AI?

The creativity of neural networks is undeniable, but when you’re making decisions that affect customers, compliance, or business operations, don’t you need something more grounded than “statistically likely”?

Maybe the future isn’t bigger models, but smarter architectures that combine the intuition of neural networks with the reliability of symbolic reasoning? With the added benefit of improved sustainability (as per my previous posts).

What Are You Seeing?

• Are your AI projects hitting similar limitations?
• Is there renewed interest in knowledge graphs and structured reasoning in your organisation?
• How are you handling the tension between AI creativity and reliability?
• Do you think NeuroSymbolic approaches are finally ready for prime time?

Genuinely curious about others’ experiences here. Massive thanks to Tony Seale , Stuart Winter-Tear Andy McMahon and many others recent posts that have inspired much of this thinking!