AI Almost Started a War. Nobody Noticed. There is something deeply uncomfortable about the way we talk about artificial intelligence. We spend enormous amounts of time discussing what AI might do ten or twenty years from now, whether machines will become smarter than humans, whether jobs will disappear, whether superintelligence will change civilization and whether robots will eventually take over some part of our lives. At the same time, some of the most important AI risks are already appearing in much less dramatic forms. They do not involve machines becoming conscious or taking control of the world. Sometimes, they simply involve a machine getting something wrong and a human believing it.
That is what makes a recent reported incident involving the
United States, China and an AI-assisted intelligence assessment so unsettling.
According to reports, an AI system was involved in producing an intelligence
assessment that incorrectly identified the contents of a Chinese vessel. The
information suggested that the vessel was carrying material associated with
China's nuclear weapons programme. The assessment was taken seriously enough
that preparations were reportedly made to intercept and board the ship.
Military aircraft were involved, and the situation was moving toward direct
action before the information was challenged and the mistake was discovered.
The phrase "AI almost started a war" is obviously
a dramatic simplification. Wars do not start because of a single computer
output. There are people, procedures, intelligence checks, military commands
and political decisions between an AI-generated assessment and an actual
conflict. But that is precisely what makes the incident worth discussing. The
AI did not need to launch a missile. It did not need to control a fighter jet.
It did not need to make an autonomous decision to attack another country. It
only needed to provide incorrect information at a moment when humans were
prepared to act on it.
That is a very different way of thinking about AI risk.
For most businesses, an AI mistake is an inconvenience. A
customer service chatbot might give someone the wrong answer. An AI assistant
might summarise a document incorrectly. A sales system might recommend the
wrong customer. A software tool might generate code containing a subtle error.
Someone notices the problem, fixes it and moves on. The consequences might be
financial or reputational, but they are usually contained.
The situation changes completely when AI enters an
environment where decisions have to be made quickly and the consequences of
being wrong are enormous. Military intelligence is one obvious example.
Financial markets, critical infrastructure, healthcare, aviation and industrial
operations can create similar problems. In all of these environments, there is
a fundamental difference between an AI system being useful and an AI system
being trusted.
And that distinction is becoming increasingly difficult to
maintain.
Modern AI systems are extraordinarily good at producing
answers that sound convincing. That is one of the reasons people enjoy using
them. They do not normally respond with a confused collection of fragments.
They give us complete sentences, explanations and conclusions. They can take an
enormous amount of information and turn it into something that looks like a
carefully considered answer.
The problem is that sounding certain is not the same as
being correct.
An AI system can be wrong without knowing that it is wrong.
It can connect pieces of information that should never have been connected. It
can interpret an ambiguous image incorrectly. It can misunderstand the
significance of a piece of intelligence. It can fill a gap in the information
with an answer that sounds perfectly reasonable. And because the final output
is presented so clearly, the person reading it may not immediately realise how
much uncertainty sits underneath it.
That becomes particularly dangerous when speed is involved.
Imagine a tense situation between two major military powers.
Something unusual is detected. Thousands of pieces of information are arriving
from satellites, aircraft, ships, communications systems and intelligence
networks. Humans cannot examine every piece of information at the same time, so
AI is used to help identify patterns and produce an assessment. The machine
produces its conclusion within seconds. The people receiving it now have
something that looks like an answer.
If that answer is correct, AI has done something incredibly
valuable. It has helped people make sense of a complicated situation much
faster than they could have done alone.
But if that answer is wrong, the speed that made AI valuable
suddenly becomes part of the problem.
The other side does not know that the information is wrong.
It sees military movements and assumes they have a reason. It responds to what
it sees. The first side then sees that response and interprets it as evidence
that its original assessment was correct. Each side begins reacting to the
other's actions, while the original mistake becomes buried underneath a growing
chain of decisions.
Nobody has to want a war for that situation to become
dangerous. Nobody has to deliberately lie. Nobody has to build a machine
designed to destroy humanity. A sequence of reasonable decisions, made using
incorrect information, can be enough.
This is why the reported incident deserves attention beyond
the world of defence and national security. It illustrates a problem that
businesses are already encountering in a much more harmless form: what happens
when people trust an AI answer simply because it sounds authoritative?
The airline industry provides a useful real-world example.
Air Canada once had a chatbot that gave a customer incorrect information about
the airline's bereavement fare policy. The customer relied on that information
and subsequently sought compensation. The case eventually reached a Canadian
tribunal, which held the airline responsible for the information provided by
its chatbot.
The details are obviously very different from a military
intelligence incident. Nobody was preparing to intercept a ship, and nobody was
worried about an international confrontation. But the underlying problem was
remarkably similar. A machine produced information that was wrong, a person
reasonably believed the information was reliable, and the organisation
ultimately had to deal with the consequences.
The lesson for businesses is not that they should stop using
AI. Quite the opposite. AI can be enormously useful when it is put in the right
place and given the right boundaries. The problem arises when organisations
confuse assistance with authority.
A customer service system can draft an answer, but
complicated or sensitive cases should have a clear path to a human employee. An
AI system used by a bank can help identify unusual transactions, but important
decisions should not depend entirely on an automated conclusion. An industrial
system can identify a potential equipment problem, but the recommendation
should be checked before someone shuts down an expensive production line. The
more serious the consequence, the more important it becomes to have someone
capable of questioning the machine.
That sounds obvious, but the pressure to automate can make
organisations forget it.
The attraction of AI is speed. Companies want fewer people
doing repetitive work, faster decisions and systems that operate around the
clock. Those benefits are real. But when a business removes the human checks
simply because the AI appears to be performing well, it can also remove the
mechanism that catches the AI when it eventually fails.
And eventually, it will fail.
This is perhaps the most important point that gets lost in
the excitement surrounding AI. The question is not whether AI will make
mistakes. It will. The question is what happens when it does, how quickly the
mistake can be detected and whether anyone has enough authority to stop the
process before the mistake becomes a real-world problem.
A good AI system, therefore, is not simply one that produces
impressive answers. It is one that operates inside a system designed to deal
with uncertainty. Important information should be traceable. High-impact
decisions should have appropriate human oversight. Unusual situations should
trigger additional checks rather than automatic action. And people using AI
should understand that a confident answer is still an answer that needs to be
evaluated.
This is particularly important because AI is gradually
moving from being a tool we consult to becoming a tool that participates in
decisions.
There is a significant psychological difference between
asking an AI, "Can you summarise this report?" and asking it,
"What should we do about this situation?" The first request is
relatively harmless. The second places the machine much closer to the decision
itself. As organisations move further toward the second category, the quality
of the surrounding controls becomes just as important as the quality of the AI
model.
That is where the conversation about AI safety needs to
mature.
We should certainly continue discussing advanced AI and the
long-term questions surrounding increasingly capable machines. But we should
not allow those futuristic discussions to distract us from the simpler risks
already sitting in front of us.
The machine does not have to become smarter than us to cause
a serious problem.
It may only have to become convincing enough that we stop
questioning it.
That is what makes the reported US-China incident so
uncomfortable. If the accounts are accurate, the danger did not come from a
machine taking control of a military system. It came from an AI-generated
assessment becoming part of a chain of human decision-making where an error
could have had extraordinary consequences.
The technology did not need to be malicious. It did not need
an objective. It did not need to understand geopolitics. It only needed to be
wrong.
And someone needed to believe it. Perhaps that is the part of the AI revolution we should be thinking about more carefully. We have become fascinated with whether machines will eventually think like humans, but an equally important question is whether humans will begin thinking less carefully because machines are doing so much of the thinking for them.
AI is going to become more deeply embedded in business, government, defence and everyday life. That is probably inevitable. The important choice is not whether we use it, but how much authority we give it and what safeguards remain around it. Because when an AI system gets a restaurant recommendation wrong, the worst outcome might be a disappointing dinner.
When it gets an intelligence assessment wrong, the
consequences can be very different.
And somewhere between those two extremes is the real AI
challenge of the next decade: learning how to benefit from machines that are
extraordinarily capable without forgetting that they can also be
extraordinarily wrong.
#AI #ArtificialIntelligence #GenerativeAI #AISafety #ResponsibleAI #AIgovernance #EnterpriseAI #Technology #RiskManagement #DigitalTransformation
