How can you trust the answers AI gives you?

With the race to scale up AI continuing apace, a natural concern arises. As AI systems become more powerful, they also seem ever more mysterious.

How can we trust systems that appear to be impenetrable black boxes, whose inner workings we only dimly understand?

One response is to try to ‘open the box’ – to identify the representations distributed across billions of weights and activations inside a neural network and to map out exactly what is being computed.

This is the active research programme of mechanistic interpretability, and it has produced genuine insights. But in general, it may prove impractical on its own.

The function of a modern deep neural network is not localised in a neat, human-readable module. It is distributed across vast stretches of the network.

Complex interactions among parameters allow the system to combine patterns and exceptions, flexibly guided by context.

That distributed structure is precisely what makes these systems so powerful and so resistant to straightforward inspection. But there is another approach.

At first glance, the opacity of AI systems seems deeply worrying. Yet on reflection, this is nothing new.

It is an all-too-familiar situation we face when interacting with other human beings.

Does AI work like the human brain?

Each of us carries around a brain of enormous complexity – around 100 billion neurons and in the order of 100 trillion synaptic connections – about which we have no direct insight.

We cannot ‘inspect’ one another’s neural circuitry. Indeed, we are no better at ‘introspecting’ our own circuits.

And yet we manage to interact successfully enough. We explain each other’s behaviour, predict responses, justify our actions, and coordinate socially well enough to get along and to sustain complex societies.

How do we do this? Not by opening the biological black box, but by engaging at the level of interaction.

We ask for reasons. We test responses. We probe with counterfactuals. We request clarification. We build trust through repeated exchanges.

In short, explanation is something that happens between agents, not something extracted directly from neural tissue.

The parallel with human interaction is instructive, but it has real limits that deserve acknowledgement.

We trust other humans in part because we share a common cognitive architecture, the same evolutionary history, and the same repertoire of embodied experience.

As such, we have reasonable expectations that other humans have goals, beliefs, and experiences roughly like our own.

When a person offers a justification for their behaviour, we might assume we can reasonably infer that it reflects, however imperfectly, something about the reasoning processes that actually produced the behaviour.

With current AI systems, that inference is far less secure. When a large language model produces a justification, it may be generating text that sounds like a faithful account of its reasoning, but bears no reliable relationship to the computational process that generated the original output.

This may appear to be a crucial difference. But decades of research in psychology suggest that ‘introspection’ by humans and AI may actually not be so different after all.

How to interact with AI more effectively

Humans, it turns out, are prolific ‘confabulators’. In other words, our brains unconsciously fill gaps in our memories with fabricated or distorted information without intending to mislead anyone.

The justifications people offer for their actions or the choices they make are often plausible-sounding narratives constructed after the fact, because they have no conscious access to the cognitive process that drove their behaviour.

What makes human interaction work despite pervasive confabulation is not that we take people's justifications at face value.

It is that we test them; we probe, challenge, cross-examine, and check for consistency over time. Interactive explainability applies these same insights to AI.

Instead of demanding full transparency of internal mechanisms, we treat the AI system as a partner in dialogue. We evaluate it through:

• Counterfactual probing (“What would you say if this assumption changed?”)
• Asking for justifications for words and actions
• Consistency checks across related responses
• Coherence performance in extended interaction, not just single queries
• Cross-examination (“but yesterday you said the opposite,” “if you believe X, how can you now be telling me Y”)

In this view, understanding is not a static property of a model’s architecture. It is an emergent property of structured interaction.

However, we should offer a note of caution here. It is entirely possible that a sufficiently capable system (whether human or AI) may pass all these checks while still being unreliable in ways that matter.

Are AI responses trustworthy?

A system that is coherent and consistent in dialogue is not necessarily trustworthy – it may actively be attempting to deceive us.

Indeed, AI-safety researchers have long interrogated AI systems using methods such as adversarial ‘red teaming’, a form of stress-test which attempts to lure the AI into forbidden or inappropriate behaviour.

What interactive explainability adds is the recognition that this kind of structured probing should not be confined to pre-deployment safety testing.

Instead, it should be a continuous feature of how we engage with AI systems.

This does not mean that internal analysis of AI is without value. Indeed, if the motivating concern is the safety of increasingly powerful systems, then understanding internal mechanisms still matters a great deal for safety, debugging, and scientific understanding.

But for practical trust, what matters most may be something more familiar: whether the system behaves coherently across contexts, responds to challenges, corrects itself when prompted, and integrates feedback over time.

Human social life already runs on this principle. We trust people not because we understand their neurons, but because they can justify themselves, respond to criticism, and maintain consistency across situations.

If AI systems can do something analogous – and if we can build infrastructure that better ensures that they do – then black-box opacity may not be as severe a problem as it first appears.

The core shift is this: explanation need not require the unveiling of hidden circuitry.

It is the achievement of mutual intelligibility through interaction, the approach that has sustained human social coordination for as long as humans have existed.

Further reading:

Working on the jagged frontier: How companies should use generative AI

What can business leaders learn from AI regulation in Ukraine?

Do you need an AI teammate?

The new AI prediction products and the risks they present

Nick Chater is Professor of Behavioural Science and a member of the Tango research project funded by the EU to explore decision-making that involves both humans and AI.

He has won numerous awards, including the Cognitive Science Society's lifetime achievement award, The Davd E Rumelhart Prize. He has served on the advisory board to the Cabinet Office Behavioural Insights Team, been a member of the UK Government's Climate Change Committee, and was resident scientist on eight series of the BBC Radio 4 show The Human Zoo. His books include It's On You, The Mind Is Flat, and The Language Game.

He teaches Behavioural Sciences for the Manager on the Executive MBA, Executive MBA (London), Global Online MBA, and Global Online MBA (London). He also teaches Judgement and Decision Making on the MSc Business and Finance, MSc Accounting and Financial Management, and MSc Accounting and Sustainability.

Simon Myers is Professor of Mathematical Genomics at the University of Oxford.

Discover more about AI and Behavioural Science. Receive our Core Insights newsletter via email or LinkedIn.