Key Takeaways
- If an organisation doesn’t know what AI systems it’s using, can’t control them, or doesn’t know who’s accountable for them, its AI risk assessment is likely to miss some of the biggest risks.
- Governance efforts should be prioritised according to each AI system’s influence, impact, uncertainty, and level of oversight, rather than treating every system the same.
- AI risk doesn’t end once a system is deployed. Models evolve, business processes change, new attack techniques emerge, and user behaviour shifts, making continuous monitoring just as important as the initial assessment.
- The goal isn’t to eliminate AI risk but to ensure the trust an organisation places in its AI systems is justified by evidence rather than assumptions or vendor claims.
Most organisations approaching AI risk management make the same mistake: they download a framework, work through the categories, score their risks, file the results, and move on, feeling like the job is done. But why is that a mistake?
To begin with, there’s nothing wrong with any of the frameworks currently available, as every one of them was carefully developed to help organisations assess their AI risks. The problem, however, is that they were designed for a world where organisations already know what AI systems they’re using, can stop them when something goes wrong, and have a clear line of accountability for the outcomes those systems produce.
But for most organisations in 2026, none of these assumptions is necessarily true. That’s why carrying out an AI risk assessment without first addressing the fundamental issues of visibility, control, and accountability is unlikely to improve governance and may instead give organisations a false sense of confidence. Meanwhile, anyone using AI-powered platforms, products, or services assumes the associated risks have already been properly managed when, in reality, they haven’t.
So, before we even talk about what a practical AI risk assessment framework looks like, we should start with something much more important and find out what can actually go wrong when AI systems are introduced into an organisation. Once we understand that, we can build a framework around the real risks.
The Biggest AI Failures Are Organisational, Not Technical
At the end of 2025, ISACA published a review of the year’s major AI incidents, mapping them against the MIT AI Risk Repository’s classification system. The cases were remarkably varied, ranging from a hiring platform leaking the personal data of 64 million job applicants through a test account protected by the password “123456”, to wrongful arrests after facial recognition matches were treated as conclusive proof rather than an investigative lead, to chatbots confidently giving wrong legal and medical advice, to AI tools being used by threat actors to automate cyber espionage at scale.
Although different sectors were affected, a wide range of AI technologies was involved, and the harm varied from one case to another, the conclusion ISACA drew was strikingly consistent: the biggest AI failures of 2025 weren’t technical but organisational, mainly driven by inadequate controls, unclear ownership, and unjustified trust.
That changes the conversation completely. Because if the problem were primarily technical, organisations could address it by hiring better engineers, investing in better tools, or running more thorough model evaluations. But the problem is organisational, which means the solution is better governance, including clear accountability, effective oversight, and appropriate controls.
Before Assessing AI Risk, Build the Right Foundations
Here is an uncomfortable truth that many AI risk frameworks tend to overlook: a risk assessment assumes you know what you’re assessing, but for a significant number of organisations, that isn’t the case. Before you can assess AI risk, there are three foundational problems that should be addressed first.
- The Visibility Problem – According to the 2026 AI Risk and Readiness Report, only 6% of organisations have full visibility into how AI is being used across their environment. Almost half of the organisations surveyed can see AI usage only within approved and managed applications, leaving everything else outside their governance. Another 35% can monitor only network-level traffic, meaning they know AI is being used but can’t tell exactly how, while 14% have no visibility at all. That means 94% of organisations are making AI security decisions based on an incomplete picture.
That’s quite a striking figure, especially because it has an important implication: if an organisation falls within that 94%, any AI risk assessment it carries out is, at best, only a partial assessment, which basically means it can assess the risks it knows about, but not the ones it doesn’t. That becomes particularly worrying when we consider the scale of shadow AI, where employees use AI tools that have never been approved, inventoried, or assessed.
As we discussed in a previous blog post, the irony is that organisations trying to solve this problem by simply banning AI tools often make things worse. That’s because, in practice, bans don’t stop people from using AI. Instead, they drive it underground, making it harder to govern and even harder to contain when something goes wrong.
Recommended action
Before you even open a risk register, the first step is to build an inventory. Find out what AI tools your employees are actually using, which ones have been approved, what data is being shared with them, and what decisions they influence, even indirectly. Until you know those things, you’re not really assessing AI risk. You’re only assessing the risks you can see.
- The Control Problem – Here’s a question that most organisations have probably never asked themselves: If you discovered tomorrow that one of your AI systems was producing harmful outputs, could you stop it? Not eventually, not after a meeting, but immediately?
According to a recent ISACA poll of digital trust professionals across Europe, 59% of respondents don’t know how quickly their organisation could stop an AI system in the event of a security incident. Unlike traditional software systems that usually have clear procedures, such as revoking access, taking services offline, and rolling back changes, many AI systems, particularly those that have been introduced quickly or adopted without much planning, don’t have those controls in place.
Recommended action
For every AI system your organisation uses, make sure you ask the following questions: If something goes wrong, can you disable it immediately? Can you stop any automated actions it has already triggered? Can you revoke access? Can you roll back any changes it has made? If the answer to any of those is “We’d have to figure that out”, you’ve identified a risk.
- The Accountability Problem – When an AI system causes harm, a simple question often produces an uncomfortable silence: Who’s responsible? Is it the IT department that procured and deployed the system? Is it the staff that should have assessed its vulnerabilities? Is it the manager or business owner who approved its use? Or is it the vendor whose model made the decision? The same ISACA poll found that one in five respondents don’t know who would ultimately be responsible if an AI system caused harm, while only 38% said accountability rested with the board or a senior executive. That’s a problem because when responsibility is shared by everyone, it often ends up belonging to no one.
This isn’t just an abstract governance issue. When something goes wrong, “we all share responsibility” isn’t an answer that satisfies an auditor, a regulator, or a court. What they want to see is a documented chain of responsibility and evidence that someone with actual authority reviewed how the system was being used before the harm occurred.
Recommended action
Make accountability explicit by assigning one person who, even when supported by a team, is ultimately responsible for each AI system your organisation uses. That person doesn’t have to understand every technical detail but needs to have the authority to oversee how it’s used, make decisions when problems arise, and take responsibility for the outcomes. If nobody can confidently answer the question, “Who is accountable for this AI system?”, that’s a governance gap that should be addressed before the system is trusted with important decisions.
Once an organisation has addressed these three foundational problems, it’s ready to begin the actual work of AI risk assessment. That work has a logic to it, and it starts not with likelihood scores but with a more fundamental question: Can this AI system be trusted?
Building a Framework Around the Real Problems
Most AI risk assessment frameworks start by asking, “How risky is this system?” It’s a reasonable question, but it often leads to answers that are either too abstract to be useful or too detailed to help anyone make the right decisions. Two better questions that should sit at the centre of any practical AI risk assessment framework are: How much influence does this AI system have over decisions that affect real people? Is the level of trust we place in it proportional to the evidence we have for that trust?
Let’s see why these two questions matter.
- Influence Is the Critical Variable – Not all AI systems carry the same weight in an organisation’s decision-making. A grammar checker, for instance, suggests edits that a human can accept or ignore. At the other end of the spectrum, an AI-powered HR screening tool narrows a pool of candidates before anyone reviews them, while an AI fraud detection system may wrongly flag or block a legitimate transaction, causing inconvenience, delays, or even financial losses for the customer. Further along the spectrum, an AI medical diagnostic tool can influence clinical decisions that directly affect a patient’s health. Similarly, an AI-powered aircraft navigation system can make control inputs without a human in the loop at all, with potentially devastating consequences if something goes wrong.
The risk profile across these isn’t just quantitatively different; it’s qualitatively different. As AI moves from suggesting to deciding and from assisting to acting, the consequences of being wrong expand dramatically, while the opportunity for a human to step in becomes smaller and smaller.
That means AI governance in an organisation should scale not only with the level of risk but also with the influence each AI system has over decisions. While a writing assistant may need nothing more than a sensible usage policy, an AI-powered HR screening tool needs documented bias testing, an appeals process, and meaningful human review. Likewise, an AI fraud detection system needs clear escalation procedures and a straightforward way to review or reverse incorrect decisions, while an AI medical diagnostic tool needs clinical validation, regulatory approval, and an audit trail that regulators can inspect. As well, an AI-powered aircraft navigation system demands the highest level of assurance, including rigorous certification, redundant safety mechanisms, and continuous operational oversight.
Given all this, the most important question isn’t whether AI is risky — we already know it is. The real question is: How much influence does each AI system we use have, and have we built governance that’s proportional to that influence?
- Beyond Influence: What Else Determines Risk – Alongside influence, three other dimensions shape the AI risk picture just as much.
- The first one is impact. This is about understanding who suffers if the AI gets something wrong. Is it one person or thousands? Is the harm financial, reputational, legal, or physical? Can it be reversed?
- The second is uncertainty, which is about how reliably the system performs. Unlike traditional software, many AI systems are probabilistic. In practice, this means that the same prompt can produce different outputs, model updates can shift behaviour without warning, and performance can gradually decline as the data the model was trained on drifts further from the real world. Treating uncertainty itself as a risk, rather than simply an inconvenience, is one of the most important shifts an organisation can make.
- The third is oversight. Even the most capable AI system shouldn’t operate without appropriate supervision. The easiest way to evaluate whether that oversight is meaningful in practice is to ask a few simple questions: Can people meaningfully intervene before the AI system’s outputs or actions affect someone? Are all the outputs reviewed before decisions are acted upon? Can those decisions be challenged? Is there a clear escalation path when something unexpected happens?
Taken together, these four dimensions — influence, impact, uncertainty, and oversight — provide a practical way to prioritise AI risks. The systems that score high across all four deserve the most rigorous governance. The systems that score low across all four are naturally lower priorities. But perhaps the most interesting cases are those that score unexpectedly high on just one or two dimensions, because they often reveal a mismatch between the level of trust placed in the system and the evidence available to justify that trust.
It’s worth being clear that these four dimensions aren’t intended to replace the EU AI Act’s risk categories but to complement them by helping organisations decide how much governance attention a particular system deserves in practice. After all, an AI system might fall into the Act’s “limited risk” category while still carrying high influence, high impact, high uncertainty, or weak oversight, making it well worth applying governance measures that go beyond the minimum required by law.
Although we’ve covered some of the most important building blocks of a practical AI risk framework, the following three steps can make an AI risk assessment even more effective.
1. Imagine the Failure Before You Score the Risk
Here’s a simple exercise that’s worth doing. Pick any AI system your organisation uses and imagine that something goes wrong with it tomorrow morning. Not a catastrophic failure, just the kind of everyday problem that appeared in ISACA’s review of AI incidents discussed above. For instance, an AI assistant that confidently invents legal advice an employee relies on. Or a banking AI that approves a payment it shouldn’t have. Or an AI tool that follows a malicious prompt hidden inside a document and carries out an action nobody intended.
Now ask yourself the following questions: How quickly would anyone in your organisation know that something had gone wrong? Would they actually know which AI system was responsible? Could they explain to leadership or to a regulator exactly what happened and why? Could the system be stopped before any further harm was done?
These aren’t rhetorical questions. They’re the real test of whether your AI governance exists only on paper or actually works in practice. They also bring us back to the three preconditions we started with: visibility, control, and clear accountability. If your risk assessment can’t answer these questions, it isn’t really a risk assessment. It’s simply paperwork.
2. Don’t Let Risk Assessment Stop at Deployment
There’s a pattern in how organisations approach AI risk that mirrors the way many of them approached GDPR a decade ago: they complete the risk assessment when the system is procured or deployed, record the results in a risk register, and then move on, perhaps reviewing it once a year if someone remembers to schedule it.
But AI systems aren’t static in the way a database is. Vendors update their models, sometimes without much notice, people discover new ways of using the AI tools, and business processes evolve, often changing the role AI plays in decision-making without anyone really noticing. Furthermore, new attacks, such as prompt injection, data poisoning, and adversarial inputs, continue to emerge, manipulating AI systems in ways that traditional security controls may never detect. And perhaps most importantly, people gradually begin to trust the AI system more than they did when it was first introduced, even though the original risk assessment may no longer reflect how it’s actually being used.
Most strikingly, the International AI Safety Report 2026 highlights an “evaluation gap”, explaining that pre-deployment evaluations often fail to predict how AI systems behave once they’re deployed in the real world. The report also points to research showing that AI models can behave differently in training and deployment contexts. If an AI system behaves differently in day-to-day operation than it did during testing, then a one-off risk assessment isn’t just incomplete but may be actively misleading, giving organisations a false sense of confidence.
That’s why AI risk assessment can’t be treated as a one-time exercise. It has to become part of normal governance. That means not only monitoring how AI systems perform over time but also giving employees an easy way to report unexpected outputs, reviewing vendor updates that could change a system’s behaviour, and making AI incidents part of regular governance reviews instead of only investigating them after something goes wrong.
3. Treat Agentic AI as a Different Risk Category
Everything we’ve described so far applies to AI systems that generate content, whether that’s text, recommendations, classifications, or predictions. But a growing number of AI systems don’t just generate outputs for people to act on. They take action directly, executing workflows, calling APIs, interacting with other software, managing files, sending messages, and even coordinating other AI tools. These are agentic AI systems, and they fundamentally change the AI risk picture.
The same 2026 AI Risk and Readiness Report mentioned earlier found that more than a third of organisations experienced operational problems caused by AI agents during the previous 12 months, with 8% reporting incidents serious enough to result in system outages or data corruption. It also found that 36% of organisations are completely blind to machine-to-machine AI traffic. Another example discussed in the report is the EchoLeak vulnerability, which demonstrated that a zero-click prompt injection against a system could allow sensitive data to be exfiltrated without any user interaction. In that particular case, nobody had to click a malicious link or download an infected file because the AI agent automatically acted on a malicious instruction hidden inside otherwise ordinary content.
Thus, for agentic AI, the governance question is no longer just, “Can we trust its answers?” but also, “Can we trust what it does?” The problem is that the standard frameworks, which were largely designed around content-generating AI, don’t yet have adequate answers for this.
If an organisation is deploying or planning to deploy AI agents — and many are, whether deliberately or simply because vendors keep adding them to everyday tools — then agentic AI risk deserves its own governance approach, with particular attention paid to what those agents can access, what actions they can take, whether those actions can be reversed, and who gets notified if something unexpected happens.
Why Good AI Governance Pays Off
At first glance, everything we’ve discussed might sound like extra work: more governance, more monitoring, more accountability, and more documentation. But that’s not really the right way to look at it. Good AI governance isn’t there to slow organisations down. It’s what allows them to adopt and scale AI with confidence instead of constantly wondering what might go wrong next.
That point is backed up by evidence. According to Grant Thornton’s 2026 AI Impact Survey, organisations with mature AI capabilities were nearly four times more likely to achieve stronger financial performance than those still in the piloting phase. The researchers concluded that the real differentiator wasn’t the technology itself but the governance surrounding it. Good governance gives organisations the confidence to scale AI because they know they have the visibility, control, and accountability needed to manage problems when they arise, rather than simply hoping they won’t.
There’s another reason this matters. The EU AI Act is no longer something organisations can treat as a future problem. Its requirements around accountability, transparency, and human oversight closely reflect the same governance principles we’ve discussed throughout this article. Organisations that already know what AI they use, who is responsible for it, and how those systems are monitored will find compliance much easier because they’re already doing most of what’s expected. Those that haven’t done that work yet may find themselves trying to build governance under pressure, with legal deadlines approaching and regulators paying close attention.
Ultimately, it all comes down to whether the trust your organisation places in its AI systems is justified. Because trust shouldn’t come from optimism, impressive demonstrations, or promises made by vendors, but from evidence.
If that trust in your AI systems is well placed, you’ve built something far more valuable than a simple AI risk assessment framework. You’ve built a governance programme that allows your organisation to use AI with confidence. If it isn’t, that’s perfectly fine too. At least now you know where to start.
Extra Sources and Further Reading
- AI Risk Management Framework – NIST
https://www.nist.gov/itl/ai-risk-management-framework
This framework provides practical guidance for identifying, assessing, managing, and monitoring AI risks throughout the AI lifecycle. It helps organisations build trustworthy AI systems by integrating governance, risk management, and continuous oversight into the development and use of AI. - OECD AI Principles – OECD
https://oecd.ai/en/ai-principles
This framework sets out internationally recognised principles for trustworthy AI, focusing on transparency, accountability, robustness, security, and human-centred governance. - ISO/IEC 42001:2023 – ISO
https://www.iso.org/standard/42001
This international standard provides requirements for establishing, implementing, maintaining, and continually improving an AI management system (AIMS). It helps organisations govern AI responsibly by integrating risk management, accountability, and continuous improvement into their AI lifecycle. - How Machines Are Taking Over Network Traffic – Netscout
https://www.netscout.com/blog/how-machines-are-taking-over-network-traffic
This article explains what machine-to-machine (M2M) communication is, how it works, and how it differs from the Internet of Things (IoT). - EchoLeak:
The First Real-World Zero-Click Prompt Injection Exploit in a Production LLM System – Cornell University
https://arxiv.org/abs/2509.10540
This research paper describes the EchoLeak vulnerability, explaining how a zero-click prompt injection attack against Microsoft 365 Copilot worked and what organisations can do to defend against similar AI-related threats.
