Key Takeaways
- AI doesn’t break GDPR, but it does expose the assumptions many GDPR concepts were originally built on.
- The hardest AI compliance problems often emerge when legal requirements meet technical realities that GDPR never explicitly anticipated.
- Several core GDPR concepts, including purpose limitation, data minimisation, transparency, and personal data, become more complex once AI enters the picture.
- Effective AI governance requires understanding not only what systems are designed to do, but also what they might do.
- The organisations best prepared for the future will combine privacy expertise, AI literacy, and sound governance practices.
Most discussions about AI and GDPR begin with the same question: how does GDPR apply to AI? While it’s a perfectly reasonable question, it may not be the one that matters most.
Despite the growing body of guidance, governance frameworks, and compliance programmes surrounding AI, many data protection professionals describe a similar experience when working through AI compliance: the tools and processes are still there, yet something feels different. Even though the assessments get completed and the requirements get met, the sense of genuine oversight we’ve come to expect never quite follows.
Which raises a more interesting question: What if AI’s most important impact on privacy isn’t that it creates new compliance challenges, but that it’s exposing assumptions that were already embedded in the GDPR itself?
That’s where I think the real story lies. Rather than forcing regulators to confront entirely new questions, AI is testing the boundaries of some of GDPR’s most fundamental concepts. Many of those concepts were developed in an environment where information was easier to trace, control, and compartmentalise than it is today. That doesn’t mean those concepts are wrong. It just means they were built on assumptions that were never subjected to the kinds of pressures modern AI systems now create.
As those assumptions come under strain, ideas that once seemed straightforward begin to look far more complex. Part of the reason is that GDPR expertise and AI literacy overlap in some important places, but the overlap is narrower than many assume.
What this means in practice is that the organisations most likely to navigate the next decade successfully aren’t those assuming GDPR privacy expertise automatically translates into AI privacy expertise, but those that understand exactly where traditional GDPR thinking still works, where it needs to evolve, and where it may be leading them in the wrong direction.
Why GDPR Built the Mental Models It Did
To understand why some long-standing GDPR assumptions become harder to apply in AI contexts, we first need to understand why they worked so well in the first place.
GDPR was designed for a world of information systems where data has identifiable attributes, processing serves a defined purpose, records follow a lifecycle, and rights can be exercised against specific information. Principles, such as purpose limitation, data minimisation, transparency, accountability, and storage limitation, were built around this reality.
AI doesn’t necessarily invalidate those principles. The challenge is that many AI systems don’t fit as neatly into the assumptions that made those principles straightforward to apply in traditional software environments. So, what AI does is weaken the connection between the GDPR framework and the reality it was designed to govern. This disconnect often goes unnoticed until someone tries to exercise a right, explain an outcome, or trace a decision. At that moment, they discover that the floor they thought was solid has quietly become glass.
Basically, AI is revealing that we may have depended more heavily on the nature of traditional software than we realised. For years, many GDPR concepts felt stable and self-evident, not necessarily because they were universal, but because the systems they were applied to behaved in predictable ways.
This is where I think the conversation becomes far more interesting. Because rather than asking how GDPR applies to AI, we should ask what happens when GDPR concepts pass through what we might call an AI translation layer. Some concepts seem to survive that translation remarkably well. But others start to behave in ways that make their original meaning harder to pin down.
The Infrastructure Problem
The most fundamental challenge in applying GDPR to AI isn’t tied to any specific article or obligation. It begins with something much more basic: what a trained model actually does.
When personal data is used to train a machine learning model, it doesn’t behave the way data behaves in traditional systems. Rather than being copied, transformed, or moved, data is used to adjust billions of numerical parameters, known as weights, that enable the model to perform its task.
As a result, the model learns from the data, and once it does that, the data itself is no longer present in a form that can be retrieved. Even though traces of data may remain, and, in some cases, be recoverable or attributable to specific individuals, models don’t store data in a traditional database format.
The practical implications of this become apparent when someone tries to exercise a GDPR right. Whether it’s a request for access, rectification, erasure, or objection, organisations often face a similar challenge because the data is no longer available as a discrete record that can be easily identified and acted upon.
By contrast, in a traditional system, this process is quite straightforward: the organisation finds the relevant records, deletes them, confirms the action, and documents the outcome. The data is gone, and the system continues to function without it.
When personal data has been used to train and AI model, responding to requests for access, rectification, erasure, or objection often means considering model retraining or machine unlearning, though alternative measures (e.g. output suppression, filtering, data source tracking, and risk-based responses) may also be used depending on circumstances. While all these approaches have their place, retraining large-scale AI models is expensive, and machine unlearning remains, especially for complex models, an active area of research rather than a mature operational solution standardised at scale.
This is where the gap between compliance theory and technical reality becomes visible. The rights framework still exists, but the mechanisms for giving effect to those rights no longer resemble the mechanisms GDPR originally assumed.
Obviously, this doesn’t mean GDPR is irrelevant in the AI context, nor does it mean the aforementioned rights cannot be exercised. What it does mean is that the operational mechanisms behind them are different from those used in traditional systems and, in many cases, aren’t yet reliably established.
Because of this, the organisations that assume the GDPR rights framework can be transferred directly into AI environments may end up building compliance programmes that look correct on paper but don’t function as intended in practice. That creates risks both for the organisations that believe they’ve addressed their obligations and for the individuals whose rights should be protected.
Four GDPR Concepts That Change Meaning at the AI Layer
The infrastructure problem is probably the deepest challenge AI creates for GDPR, but it’s far from the only one. Once you start looking closely, several familiar GDPR concepts begin to shift in subtle ways. Below are four principles where AI introduces a layer of complexity that only becomes visible when we look beyond the legal language and examine the operational reality underneath it.
1 – Purpose Limitation and the Indeterminate Model
Under Article 5(1)(b) of GDPR, purpose limitation requires personal data to be collected for clearly defined and lawful purposes and not subsequently used in ways that conflict with those purposes.
In traditional systems, this principle is relatively straightforward to apply. An organisation collects data for a defined purpose, documents that purpose, restricts how the data is used, and can demonstrate compliance through policies, controls, and audits. The purpose acts as a genuine boundary around what the system does.
AI, however, makes that boundary more difficult to define once you move beyond the simplest deployment scenario. That’s because a model can be trained for one purpose, fine-tuned for another, and later integrated into products that the original developers never anticipated. Even though at the fine-tuning stage, a further purpose can be specified, and a compatibility analysis can be conducted under Article 6(4), once a model is made available through an API or embedded into a broader ecosystem, its eventual uses are often shaped by whoever builds on top of it rather than by the organisation that originally trained it.
This creates an uncomfortable question: What exactly is the purpose that should be limited? Many organisations still document AI purposes in DPIAs and governance frameworks as if they were dealing with traditional software. But those descriptions often reflect the intended use case rather than the full range of capabilities a model makes possible. As a result, purpose limitation currently functions more as a documentation exercise than as a meaningful constraint on how the system will be used.
2 – Data Minimisation and the Capability Trade-Off
Article 5(1)(c) of GDPR requires that personal data be adequate, relevant, and limited to what’s necessary in relation to the purposes for which it’s processed. For experienced privacy professionals, the instinct toward collecting only what’s genuinely needed is almost second nature, and it remains just as important in AI systems as it does in traditional ones.
The challenge is that AI introduces a tension that many privacy frameworks were never designed to address. In traditional systems, collecting less data is usually a straightforward win from a privacy perspective. But in AI, reducing the amount or diversity of training data can also reduce the capability of the system itself. In general, models trained on larger and more varied datasets tend to perform better, which means that data minimisation decisions often become decisions about how much capability an organisation is willing to sacrifice in order to reduce privacy risk.
As expected, that changes the conversation in an important way: while the traditional GDPR question is whether the data is necessary for the purpose, in AI, necessity is rarely a simple yes-or-no question. Since more data may improve performance and less data may reduce privacy risk, deciding where to draw the line isn’t simply a compliance decision but a governance decision that requires organisations to balance capability, risk, and accountability.
3 – Transparency versus Explainability
According to Article 5(1)(a) of GDPR, organisations are required to explain what data they collect, why they collect it, how it’s used, who it’s shared with, and what rights individuals have.
AI complicates that picture because an organisation can be completely transparent about the fact that it’s using AI, where the data came from, what the system does, and how individuals can exercise their rights, while still being unable to explain why the system produced a particular output.
This is where the distinction between transparency and explainability becomes important. Transparency tells us what a system is doing, while explainability tells us why a particular outcome occurred. Many AI systems, particularly deep learning systems with complex architectures, produce outputs through processes that may be difficult to fully reconstruct in a human-intuitive way.
This issue becomes particularly important under Article 22 when AI systems are involved in decisions that affect individuals. This article establishes a general prohibition, subject to exceptions, on decisions based solely on automated processing where those decisions produce legal or similarly significant effects. Where such decisions are permitted, individuals are entitled to obtain human intervention, express their point of view, and contest the outcome.
Applying these safeguards in practice becomes difficult when the system’s reasoning cannot be reconstructed in a way humans can readily understand. This creates a tension: on the one hand, individuals need enough information to understand and challenge decisions that affect them; on the other hand, organisations aren’t required to disclose the full technical workings of a model in order to provide meaningful information about the logic involved.
4 – Inferred data and the Expanding Boundary of Personal Data
Article 4(1) of GDPR defines personal data as any information relating to an identifiable person, such as names, email addresses, purchase histories, and health records. AI complicates that picture because it can generate information about people that was never explicitly collected in the first place.
For instance, a model trained on browsing behaviour, purchasing patterns, location data, or other seemingly ordinary information may be able to infer health conditions, political views, financial vulnerability, personality traits, and even likely future behaviour with a level of accuracy that many people find surprising. In other words, even though the data going into the system may appear relatively harmless, the information coming out of it may be highly sensitive.
This isn’t entirely new territory. GDPR’s definition of personal data has always encompassed inferred data, and the prevailing regulatory view, reflected in EDPB guidance, is that Article 9’s special categories apply to inferred information as much as to information collected directly.
But there is a gap between what the law clearly covers and how consistently organisations implement it in practice. Most privacy assessments and data inventories focus on what an organisation deliberately collects, stores, and shares. Far less attention is paid to what an AI system might infer from that information once it starts identifying patterns that humans can’t easily see. As a result, organisations can end up documenting the inputs while paying much less attention to the outputs.
This is partly an awareness problem, partly a technical complexity problem, and partly a governance problem: the people conducting DPIAs may not know enough about how an AI model works and what it’s capable of inferring, while the people who understand the model may not be involved in the privacy assessment process. And that gap is where some of the most significant future privacy and regulatory risks are likely to emerge.
Beyond Compliance: The Governance Question AI Forces Us to Ask
This is probably the most uncomfortable part of the conversation, but it’s also the part that many discussions about GDPR and AI never really address.
For years, privacy programmes have been built around a fairly straightforward idea: identify the risks, apply the relevant controls, document your decisions, and demonstrate compliance. In many situations, that approach works well because the systems being assessed are relatively predictable. You can usually understand what the system does, what data it uses, and what the likely consequences will be.
AI makes that process less straightforward because AI systems can create outcomes that only become visible after deployment. A model may be used in ways its developers never anticipated, may identify patterns nobody expected to find, and may influence decisions in ways that are difficult to detect until people start interacting with it at scale.
That’s where governance starts to matter. Because good governance isn’t just about documenting what a system is supposed to do but also about understanding what it might do, what assumptions sit behind it, where things could go wrong, and who is responsible when they do. Those questions are harder to answer than compliance questions because they require judgement rather than checklists.
There’s also a practical reason why this matters. Most users don’t care whether an organisation completed a DPIA, updated a privacy notice, or identified a lawful basis. What they care about is whether the system behaves in a way that feels fair, reasonable, and trustworthy. When it doesn’t, confidence is lost very quickly, regardless of how much compliance documentation exists in the background.
To me, that’s what AI is revealing most clearly: the difficult part isn’t always applying the rules but understanding the systems well enough to know whether the answers those rules produce are actually leading us in the right direction.
Where This Leaves Us
GDPR remains extremely valuable in the AI era. The principles around accountability, transparency, individual rights, and risk assessment aren’t suddenly becoming irrelevant just because AI has arrived. If anything, they’re providing much of the foundation that organisations need in order to build responsible AI governance.
The challenge is that AI creates forms of risk, uncertainty, and inference that don’t always fit neatly within the assumptions GDPR was built on. Thus, the organisations that are likely to handle this well won’t be the ones that treat GDPR as a complete solution to every AI problem. Nor will they be the ones that dismiss privacy and data protection as yesterday’s concerns. They’ll be the ones that recognise that effective AI governance isn’t just about compliance but also about understanding how the technology works, what risks it creates, and where human judgement matters.
That’s not always an easy balance to strike. It asks people to think about technical realities, legal requirements, and governance questions at the same time. But that’s also what makes this area so interesting. After all, some of the most important conversations about AI are happening right where these three worlds meet.
Extra Sources and Further Reading
- Does Training on Synthetic Data Make Models Less Robust? – Cornell University
https://arxiv.org/html/2502.07164v1
This paper examines whether synthetic AI-generated training data makes language models less robust by reinforcing their existing weaknesses. - On the Opportunities and Risks of Foundation Models – Cornell University
https://arxiv.org/abs/2108.07258
In this paper, the authors introduce the concept of foundation models and argue that these models are trained on broad data and then adapted to a vast range of downstream uses that are often unknown at the time of development. - Algorithmic Transparency and Explainability under the GDPR – Diva Portal
https://uu.diva-portal.org/smash/get/diva2%3A1941957/FULLTEXT01.pdf
This study explores the practical and legal challenges of providing meaningful explanations for complex AI systems under GDPR’s transparency requirements. - Opinion 28/2024 on Certain Data Protection Aspects Related to the Processing of Personal Data in the Context of AI Models – EDPB
https://www.edpb.europa.eu/system/files/2024-12/edpb_opinion_202428_ai-models_en.pdf
This paper examines how core GDPR principles, such as purpose limitation, lawful basis, transparency, and data subject rights, apply to modern AI models, offering practical guidance on some of the most challenging compliance questions raised by AI.
