Key Takeaways
- In an AI system’s lifecycle, responsibility, accountability, and practical control are typically spread across the provider, vendor, deployer, customer, and any other third parties involved.
- The person legally designated as the provider or deployer of an AI system may not be the person who had the practical ability to identify, understand, or prevent a particular failure.
- Having a person technically able to override an AI system isn’t enough if they lack the knowledge, training, authority, or practical ability to challenge its recommendations.
- Monitoring, logging, documentation, and clear ownership of changes are essential because they make it easier to understand what happened and establish responsibility when an AI system produces a problematic outcome.
- The key question isn’t who owns the AI, but who controls the risk. Following that question across the system’s lifecycle provides a more realistic picture of accountability than relying on any single legal or organisational label.
What happens when an AI system makes a decision that’s wrong, but nothing about the system appears to be broken?
To see how this can happen, let’s look at a simple scenario: an applicant is unfairly rejected for a job, yet the AI tool that helped assess the application hasn’t crashed, there was no error, and no one can say exactly why the decision was made. The tool has simply done what it was designed to do, according to the rules, configuration, and data it was given.
Except that something has gone wrong. Somewhere along the way, the system has gradually started filtering out qualified candidates from a particular group. Perhaps a change in the data has altered its behaviour. Or maybe someone has changed the configuration in a way the provider never anticipated. Or the person who was supposed to oversee its decisions has become so accustomed to trusting its recommendations that, over time, human oversight has existed mostly on paper.
Eventually, the problem was noticed. But by then, there was quite a long chain of people involved in the process. If we were to ask who was responsible for what happened to all those applicants, who do you think should be held accountable? The developer who built and trained the AI agent? The vendor who integrated it into the company’s hiring process? The employee who bought it? The one who configured it? Or the hiring manager who accepted its recommendation?
But what if we asked each of the people mentioned above? We would most probably get five different answers, with each one pointing somewhere else. The developer might say they built a general-purpose model and cannot be held responsible for how the customer configured or used it. The vendor might say they simply integrated the model according to the specifications they received. The employer might say they purchased a compliant product and had no realistic way to inspect the model’s internal workings. The hiring manager might say they were simply following the recommendation the system was designed to produce. The problem is that none of them believes they’re responsible. And that’s precisely what makes accountability so difficult.
Some may say that, if we want to determine who should be held responsible, we should look at the EU AI Act. But that only takes us as far as establishing which actors fall into legally defined roles, such as provider and deployer, and what obligations those roles carry. It doesn’t necessarily tell us who can be held accountable when an AI system causes harm. In those cases, the question is rarely just about who had a regulatory obligation. The more important question is who had the power to see the risk, understand it, and stop it before it reached the person on the other side of the decision.
Furthermore, the Act doesn’t tell us what happens when a carefully worded contract between a vendor and a customer says one thing, while the reality of how the system was deployed, configured, and then used tells us something else.
Thus, the real difficulty with AI accountability isn’t that nobody is responsible for anything, but that responsibility, accountability, and control end up spread across different people, with each of them holding only a piece of the picture, while the consequences of the decision fall on nobody in particular.
For instance, an organisation may have formal responsibility for an AI system’s output while having no control over the model that produced it. A provider may have technical control over how a model behaves while knowing very little about the context in which a customer eventually uses it. And a human may technically have the final say over an AI-assisted decision while having very little practical ability to question or challenge the system, making it difficult to argue that they’re truly “in control”.
Once you stop asking “Who is responsible?” and start asking “Who controlled the risk at the point where it could actually have been stopped?”, the problem above stops looking like a puzzle with a missing answer and starts looking like what it really is: a chain of decisions in which each actor controlled a different piece of the outcome.
That’s because responsibility, accountability, and control aren’t some fixed labels attached to particular roles but something that can shift, overlap, and accumulate as an AI system moves from design and development through deployment and use, all the way to the moment something goes wrong. So, perhaps the clearest way to understand how this chain of decisions actually works is to follow an AI system through its various stages.
In the rest of this blog post, we’ll do exactly that, looking at what happens at each stage and how responsibility, accountability, and control can change as the system moves through its lifecycle and across organisational boundaries.
Design and Development
At the earliest stage, the answer to “who controls the risk?” is quite straightforward. The provider controls the architecture, the training process, the testing regime and, to a large extent, the limitations that will later be inherited by anyone using the system.
For example, if an AI recruitment model systematically performs poorly when assessing candidates from a particular educational background because those candidates are inadequately represented in the training data, the provider is clearly part of that story, because the flaw was introduced during development, when the provider was in the best position to identify and address it.
But even here, the apparently simple answer begins to become more complicated once we consider how AI systems are actually used. A provider developing a general-purpose AI model can test the model against particular datasets, scenarios, and benchmarks, document known limitations, and take considerable care over the development process only to find out that a customer has placed the system into a context that was never anticipated during development.
This distinction matters because it shows why provider responsibility can’t simply be extended indefinitely downstream. The responsibility for risks introduced during the design and development phases is real, but it has a boundary, and that boundary lies exactly where the provider’s knowledge of eventual context runs out.
This is where the first crack in the idea that one person can simply be “responsible for the AI” begins to appear.
Procurement and Integration
This is where the “we only bought the AI” argument begins to look less convincing. When an organisation chooses an AI tool, it’s already making decisions that shape everything downstream. At that moment, it also decides what information it needs before signing the contract and, at least in principle, it understands the limitations it’s prepared to accept and the safeguards it will need to put in place.
Then comes integration and configuration, which can mean fine-tuning the model based on the organisation’s own data, connecting it to an internal database, configuring it, and adding rules meant to override outputs in certain situations. Each of these represents a point at which the organisation can change how the system operates and a point at which potential future AI risks can be addressed.
The EU AI Act recognises this in Article 25, which sets out the circumstances in which a deployer that substantially modifies a high-risk AI system can be considered its provider and become subject to the associated obligations. The practical implication of this is worth considering because, when an organisation substantially modifies an AI recruitment system or any other high-risk AI system for that matter, “we’re just the customer” may no longer be a sufficient defence.
But even when the organisation accepts that it has a role in shaping the system, another problem can emerge: responsibility debt. Much like a codebase accumulates technical debt, an organisation can accumulate responsibility debt when questions about ownership, oversight, and decision-making are left to be dealt with later.
Every unresolved question at procurement and integration — including questions relating to who owns the training data relationship, who is authorised to approve future configuration changes, who monitors the system after go-live, and who investigates if something goes wrong — represents a gap in the responsibility structure, which basically means something that exists on paper but has no actual owner.
Just as with technical debt, responsibility debt can remain invisible while the system continues to work as expected. However, they can quickly become visible the moment something goes wrong, and the organisation may discover that the questions everyone assumed someone else was responsible for answering were never actually assigned to anyone.
Deployment and Human Oversight
This is where I think the language around responsibility becomes particularly misleading, because “the AI made a bad decision” quietly erases the human and organisational choices that turned a statistical output into an actual consequence for a real person. The more accurate question isn’t what the AI decided but who created the conditions under which the AI’s recommendation became the decision.
Whenever concerns about automated decision-making are raised, the answer often seems to be that there must be a human in the loop. In reality, a person nominally able to override a system’s output isn’t the same thing as a person exercising meaningful oversight, and the gap between the two is exactly where much of the real risk in AI deployment lies.
Meaningful oversight requires that the person actually understands what the system can and can’t do, that they have been trained to recognise when an output looks wrong, and that they have the authority to reject recommendations. For example, a hiring manager reviewing one hundred AI-scored applications under pressure to clear the queue may technically have the power to override the system, but if challenging its recommendations requires a lengthy justification each time, how meaningful is that oversight in practice?
The AI Act itself recognises this distinction. Article 14 requires high-risk AI systems to be designed and developed so that they can be effectively overseen by humans, while Article 26 places operational responsibilities on deployers to assign people to carry out human oversight and ensure they have the necessary competence, training, support, and authority to override the system’s output when necessary.
This is where the Fundamental Rights Impact Assessment obligation for certain high-risk deployments becomes particularly important, because it forces exactly the questions a provider may not be able to answer fully on its own: Which affected groups face which risks in this specific context of use? The provider can know the technology in depth, but only the deployer knows the context in which the technology will actually be used. In real-world situations, AI-related harm tends to happen precisely because those two types of knowledge don’t meet.
Monitoring
Because responsibility doesn’t end once an AI tool is approved and switched on, treating monitoring as an afterthought rather than as part of the accountability chain is one of the most consequential blind spots in how organisations approach AI systems.
An AI system can become problematic without anyone changing a single parameter, simply because the context in which it’s being used has changed, or the data it now receives no longer resembles what it was trained on, or its outputs have quietly drifted in a direction nobody has been watching for.
As a result, an AI tool that performed acceptably when it was first deployed can behave very differently months later not only from what the organisation expected but also from how it behaved during testing and immediately after launch.
That raises a different set of questions: Who is watching? Who controls the risk? Who actually has visibility into the system’s ongoing performance and the authority to act on what they see? If nobody has that role in a meaningful sense, that gap is itself the failure, discovered only in hindsight.
This is why monitoring should be part of the accountability chain itself. If an organisation has the ability to see that a system is behaving differently but no specific person has the authority to step in, make changes, or even stop the system, then the organisation may have visibility into the risk without having meaningful control over it.
Modification
When it comes to AI systems, rarely does anyone sit down one morning and decide to make changes. Changes often happen gradually, as someone adjusts a threshold because the results don’t seem quite right, retrains the model with newly collected data, or extends its use to a new case the system was never validated against.
Maybe each change seems small and perfectly reasonable but over time, multiple changes can gradually alter the way the system operates or the purpose for which it’s being used, and so the organisation may eventually find itself using something different from what it originally purchased.
Under Article 25, this matters a lot because, in certain circumstances, a substantial modification to a high-risk AI system can mean that the organisation is treated as its provider and becomes subject to the corresponding provider obligations. The important point here is that nobody necessarily has to make a conscious decision to change the organisation’s legal position. The changes to the system and its use can happen first. To sum all this up, we can say that when an AI system or its use changes, so should the responsibilities that come with it.
And this is another reason why the responsibility debt I mentioned above is dangerous. If nobody has been given ownership of changes after deployment, an organisation can gradually move into a different legal and operational position without anybody explicitly deciding to do so, or even worse, realising that it has happened.
Failure
This is the point the earlier stages have been building towards: the moment when something goes wrong and someone asks “Who is responsible?”.
The first problem, however, may not be legal at all, but one of evidence. Let’s take our initial example, in which an AI system harmed several job applicants by incorrectly rejecting their applications. In this case, the questions that matter are concrete: Which version of the model was running? What configuration was active at the time? What input data contributed to the output in question? Was a human reviewing it? What information did the hiring manager have when they made the decision? Has the system been modified since its last review? Was it being used for the purpose for which it had originally been assessed?
Without answers to these questions, we can’t reconstruct what happened, meaning that assigning responsibility becomes considerably more difficult, even when the responsible actors appear obvious on paper. That is where logging and documentation become essential. They aren’t simply compliance paperwork alongside accountability but what makes accountability possible in the first place. The AI Act itself recognises their importance by requiring certain events to be logged for high-risk AI systems. After all, you can’t meaningfully ask who controlled a risk if you can’t establish what happened.
But establishing what happened, and even who is responsible, is only part of the problem. At the same time, the parties may already have agreed between themselves how responsibility is allocated. And that leads to the second problem: contracts won’t settle the matter the way people often assume they do. That’s because a contract between the vendor and the customer can allocate obligations, and it may even state that the customer bears responsibility for certain outputs. But it can’t simply rewrite the regulatory position created by the actual facts of who built, modified, and deployed the system, nor can it determine who a harmed third party may be entitled to pursue.
In this regard, three separate questions are worth keeping apart in any post-incident review: What does the contract say? What does the regulation say? Who would actually end up liable if a harmed person brings a claim? Those three answers aren’t guaranteed to match, and assuming they do is one of the most expensive mistakes an organisation can make.
That third question exposes the biggest structural gap of all, which is that AI Act compliance, civil liability for actual harm, and data protection are governed through separate legal frameworks that don’t map neatly onto one another. The AI Act establishes the legal obligations for actors like providers and deployers, but it doesn’t by itself determine who can recover damages when an AI system causes harm.
Civil liability may instead depend on the applicable national liability rules or, where relevant, the EU’s product liability framework. But there is no single EU-wide liability regime covering all forms of AI-related harm. Alongside these liability rules, the GDPR adds another layer when personal data is involved, with its own distinction between controllers and processors.
In other words, an organisation can be a fully compliant deployer under the AI Act, correctly identified as a controller under the GDPR, and still have no real clarity about who bears civil liability if the AI system it operates causes harm to someone.
Who Controls the Risk
Nearly everything written about AI accountability, including everything above, focuses on the chain of actors: provider, vendor, deployer, operator. The person actually affected by the decision barely appears in these discussions, being mostly treated as the endpoint of the diagram rather than a participant in it.
But the rejected candidate doesn’t experience “provider responsibility” or “deployer responsibility”. They experience a decision they can’t see inside or fully understand, reached through a chain of choices stretching back through configuration, integration, procurement, and design.
The question that actually matters to that person isn’t an abstract one about regulatory categorisation. It’s simply: “Who can I actually contact, and can they actually do anything?”
That is arguably the clearest test of whether an organisation’s responsibility structure exists in practice or only on paper. The test isn’t whether every role has been correctly labelled under the AI Act or whether the contract with the vendor is watertight, but whether there’s a genuine, identifiable person inside the organisation who can receive that question and give the applicant an answer that means something.
And that brings us back to the question at the heart of this article: Who controls the risk? More importantly, who controls the risk at every point where the system can fail, and is anyone actually positioned to catch it before it does?
When applied across the lifecycle of an AI system, from design to the moment of failure, these questions produce a far more honest picture than any single role or label ever can. Together, they force an organisation to look at the gaps between formal responsibility, practical control, and who ultimately has to answer for the outcome, rather than assuming those three things automatically line up.
Because they very often don’t. And recognising that is the first step towards actually closing the gap, rather than discovering it for the first time when something has already gone wrong.
Extra Sources and Further Reading
- Advancing Accountability in AI: Governing and Managing Risks Throughout the Lifecycle for Trustworthy AI – OECD
https://www.oecd.org/en/publications/advancing-accountability-in-ai_2448f04b-en.html
This paper explores how organisations can build accountability into AI governance throughout the AI lifecycle. It examines how risk management, clear responsibilities, and traceability can help organisations identify and address AI risks more effectively. - AI Risk Management Framework (AI RMF) – NIST
https://www.nist.gov/itl/ai-risk-management-framework
The NIST AI Risk Management Framework provides practical guidance for organisations to identify and manage AI risks throughout the system lifecycle. It focuses on building trustworthy and responsible AI through governance, risk assessment, measurement, and ongoing management. - EU AI Act, Article 14: Human Oversight – European Commission
https://ai-act-service-desk.ec.europa.eu/en/ai-act/article-14
Article 14 sets out the EU AI Act’s requirements for effective human oversight of high-risk AI systems, including the ability to understand, monitor, interpret, and override system outputs. It is particularly relevant to the distinction between nominal and meaningful human oversight discussed in this article. - Recommendation on the Ethics of Artificial Intelligence – UNESCO
https://www.unesco.org/en/articles/recommendation-ethics-artificial-intelligence
This recommendation provides a global framework for the ethical development, deployment, and governance of AI, with a strong focus on human rights and human oversight. It also emphasises responsibility and accountability throughout the AI lifecycle, including the need for auditability and traceability.
