Key Takeaways
- AI risk doesn’t sit inside the AI alone. The environment, data, people, processes, decisions, and controls around an AI system all influence the risks it creates.
- Context can change the level of risk without changing the AI system itself. The same system can present very different risks depending on where it operates, how it’s used, and what decisions it influences.
- Organisations don’t stand still, so AI risk can change over time. Use cases expand, systems are connected to new data, and tools are repurposed, potentially moving the “sheep” and “sheepdogs” into a very different part of the field.
- Governance is more than policies, controls, and compliance. It also requires clear ownership, active oversight, and people who can recognise when circumstances have changed and intervene when necessary.
- Organisations can use five simple questions to keep AI governance grounded in reality: Why are we doing this? What is the system actually doing? Where is it operating? Who owns it and has the authority to act? When do we intervene?
Why Putting a Boundary Around AI is Not Enough
When it comes to AI governance, there’s something quite appealing about the idea of putting a fence around AI. Things like laws, regulations, policies, standards, risk assessments, and compliance frameworks currently form a sort of “fence” we use to try to keep AI within boundaries and also understand what we should or shouldn’t do. The EU AI Act is one example of this approach, with different obligations depending on the risk category and intended use of an AI system. And, of course, these boundaries matter a lot.
But there’s a problem with relying on a fence alone. Because, just like in real life, a fence can’t govern what’s happening inside it, regardless of how good it is. This is the idea behind the rather unexpected analogy between AI governance and the farm you’re seeing in the image above.
Why is this relevant to AI governance? Well, the instinct in most organisations is to reach for exactly that kind of fence, which usually means drafting a policy, running a risk register, signing off on an acceptable use document, and then everybody can breathe a little easier because now there are rules. And I can see why organisations are drawn to this. After all, rules are visible and auditable, and they’re easy to present as evidence that governance is in place.
But, as I already mentioned, a fence is just a boundary, not a way of controlling the activity within it. Just like a farmyard fence, which tells us where the field ends but says nothing at all about what happens to the sheep and sheepdogs inside it, a “fence” around an AI system doesn’t tell us what happens to the data the system uses, how the system is actually used, who is watching it, or what changes when it’s exposed to different environments, either inside or outside the organisation. This last point turns out to matter far more than it often gets credit for when we assess AI risk.
Where Does AI Risk Actually Come From? It’s Not the Sheepdog, It’s the Field
One of the things I find most interesting is how easily we end up looking for risk inside the AI itself. At least, that’s what many professionals interested in adopting AI do, with most of them focusing on whether the model is accurate, secure, and compliant, and whether there are appropriate measures in place to address the risks associated. While these are all sensible considerations, they don’t necessarily tell us what the risk will look like once the technology is being used in daily operations.
This aspect is important because risk isn’t only inherent to AI but can also come from what an organisation does with the technology. For instance, an AI system can be deployed in a different environment from the one it was originally designed for, used in a different process or by a different team, or configured to influence decisions much more heavily than initially intended. In all these situations, the technology may not have changed at all, but the context has.
Risk, therefore, is the sum of different variables that come into contact with or somehow influence an AI system. Coming back to our analogy, we can say that risk emerges from all the interactions between the sheep, sheepdogs, farmer, farmhand, the type of field, the fence, and everything happening inside and even outside the farm. By “outside the farm”, I mean everything beyond the immediate organisational environment that can influence an AI system, including other departments, systems, and even external environments or organisations with different levels of governance.
To better illustrate the importance of the environment on AI behaviour and related risk, let’s go through a small thought experiment. Take the same flock, together with all the sheepdogs from our analogy, and put them in three different places. First, a farmyard, walled and gated, where the worst that can happen is a scuffed fence post. Second, a gentle hillside, open and grazeable, where the flock and sheepdogs can wander further than intended, but the ground is still forgiving. Third, the edge of a cliff, where one wrong step can now end in something terrible that we can’t undo.
Nothing about the sheep or sheepdogs changed across those three scenes. What changed was the field. And this, I think, is one of the most overlooked ideas in AI governance, because we’ve become very good at focusing on and interrogating the model, but not nearly as good at interrogating the environment it has been let loose in.
While the infographic gets something important right by placing AI risk within a wider environment, I want to push this a step further, even beyond the environment — which, after all, is only a snapshot of the situation at a particular moment — and highlight the fact that organisations don’t sit still either.
The organisation into which a system was deployed rarely remains the same six months later, either because it may expand its use of the AI system, adopt new data sources and connect the system to them, or develop new departments and repurpose the system for those departments.
While each of these moves might look like a sensible bit of scaling, the “sheep” along with the “sheepdogs” can gradually be moved from the farmyard to the hillside and then to the cliff edge without anyone drawing a new fence. And that’s because nobody thought of it as a new field at all, but just an extension of the old one.
This is the part of the AI risk conversation that a one-time risk assessment is unlikely to catch. A risk assessment carried out today tells us about the current condition of the farmyard, but tells us nothing about the cliff edge the system may be standing on six months later. Which is exactly why AI governance has to behave less like a certificate we file away and more like a habit we keep returning to, in the same way a farmer and farmhand return to the same fences and gates every day to check on the flock and sheepdogs and keep an eye on what’s happening, knowing that the sheep and sheepdogs can move to another part of the field at any time, where the conditions might be completely different.
How That Risk Actually Gets Governed
If risk lives in context rather than in the AI model alone, then governance has to be built around the same idea, and this is where the farm’s characters earn their keep, because each one represents a different piece of the picture.
The AI Farm Characters
The Sheep
To begin with, the sheep are the data. They have value, which is precisely why they need to be protected and governed. The sheepdog is there to manage the flock, but the flock isn’t simply a passive part of the farm. What happens to it matters. For instance, if the sheep wander outside the fence, they’re no longer within the boundaries that were supposed to contain them. Additionally, the way the sheepdog interacts with the data can create risks of its own.
The Sheepdog
The sheepdog is the AI itself. It has been trained by humans, but once it’s let loose in the field, it can act with a degree of autonomy. And that matters because the fact that humans trained or configured it doesn’t mean they’re directing every action it takes. Like a sheepdog, the AI can be given a purpose and certain boundaries, but what it actually does in a particular situation can depend on the environment it finds itself in.
The Farmer
The farmer represents leadership and accountability. They’re not the person doing the day-to-day work, but the person responsible for setting the purpose, deciding why the AI is being used in the first place, and ultimately taking ownership of what happens. In an organisation, this is the part that can easily become unclear, especially when responsibility for AI is spread across different people and teams. There may be plenty of policies, processes, and controls in place, but if nobody has clear ownership of the decisions being made, it becomes difficult to know who is actually accountable when something goes wrong.
The Farmhand
The farmhand represents the practitioner responsible for the operational work. They’re the person closest to the ground, responsible for implementing the system, supervising how it’s used, and making sure the people working with it understand how it should be used. They’re also the person who should be asking the simple question of whether a particular AI system is still serving its intended purpose, and whether that use is still justified. That question can easily get skipped once a tool has become part of everyday operations, because by the time anyone comes back to review it, the focus is often on whether it’s technically compliant rather than whether it still serves the purpose it was originally introduced for.
The Fence
The fence represents the controls, rules, policies, boundaries, and compliance requirements that are put in place around the AI system. Unlike every other character on the farm, though, the fence doesn’t do anything; it doesn’t watch, decide, or act. It just marks where something starts and ends. Which is exactly why governance can’t stop there.
The Farm
And that brings us to the farm itself, which represents the organisation and the operating environment in which all of this takes place. The farm matters not only because none of the other characters exists in isolation but also because it determines how much risk is actually present.
Finally, the whole farm system is governance. It isn’t just the farmer, the farmhand, the sheep, the sheepdog, and the fence, but the way all of these elements work together. If the sheepdog disappears out of sight, or the sheep moves beyond the fence, that tells us something about how well the system is actually being governed. That’s why governance can’t be reduced to only having the right policies, controls, and responsibilities in place, but it also has to account for what’s actually happening across the whole farm and how that changes over time.
This is precisely where the human-in-the-loop concept fits into the story. The important point here is that there’s a meaningful difference between having a human somewhere in a process and having someone who is actually paying attention. A human sitting in front of a screen and clicking approve on outputs they have stopped genuinely scrutinising isn’t real oversight. Also, oversight can’t be something that happens once and is then considered done. It has to move with both the system and the environment, meaning the human has to actively watch what’s happening and be capable of “redirecting the flock and sheepdogs” before they reach the edge, rather than simply watching them get there.
So the question worth asking about any human-in-the-loop control isn’t whether a person is present, but whether that person could actually stop the flock and sheepdogs from reaching the cliff if they needed to, and whether anyone would notice if they didn’t.
The Question Worth Keeping
Put together, none of these characters is sufficient on their own, and that’s really the point. A fence without a farmhand gives us regulation without active oversight. A farmhand without a farmer gives us oversight without anyone ultimately being accountable for what it finds. And a farmer without the farm system gives us authority and accountability but without the means to put that authority into practice.
The version of this that I keep coming back to is quite simple, and it comes down to the five core questions journalists use to gather information, which we can apply to any AI use case: Why are we doing this? What is the system actually doing? Where is it operating? Who owns it and has the authority to act? When do we intervene?
Answering all these questions honestly gets us much closer to real governance than any fence, however well built, can give us on its own. Because AI governance has never really been about building a stronger perimeter around the technology. It’s about understanding the environment an AI system is operating in, paying attention to how people use it and how that use changes over time, being clear about who is accountable, and being honest about whether anyone is actually holding the crook, ready to move, rather than simply leaning on the fence and hoping the sheep and sheepdogs behave.
Extra Sources and Further Reading
- The Concept of Accountability in Artificial Intelligence Gaps in Current Legal Frameworks – Research Gate
https://www.researchgate.net/publication/395305538_The_Concept_of_Accountability_in_Artificial_Intelligence_Gaps_in_Current_Legal_Frameworks
This academic article examines how AI complicates traditional ideas of accountability, particularly when it comes to identifying who is responsible for decisions and harms involving AI. It looks at gaps in current legal frameworks, including problems with attribution when multiple actors are involved in developing, deploying, and operating AI systems. - AI Risk Management Framework – NIST
https://www.nist.gov/itl/ai-risk-management-framework
This source provides a foundational framework for managing AI risk across the design, development, deployment and use of AI systems. It’s particularly useful for your arguments about governance being continuous and involving multiple organisational roles. NIST explicitly treats risk management as something that should happen throughout the AI system lifecycle. - AI Principles – OECD
https://www.oecd.org/en/topics/ai-principles.html
This source sets out the OECD AI Principles, an international standard for the responsible development and use of AI. It covers human oversight, transparency, accountability, safety and the ongoing management of AI risks throughout the system lifecycle. - EUR-Lex – European Union
https://eur-lex.europa.eu/eli/reg/2024/1689/2026-07-27/eng
The EU’s legal framework sets out the requirements for managing the risks associated with high-risk AI systems, including continuous risk management throughout their lifecycle and post-market monitoring.
