Skip to main content
§ Video

Your AI rollout will fail without training — and the EU AI Act now says so

Watch on YouTube ↗

AI tools are spreading through every department of every organisation. That looks like progress. It is also how organisations acquire a category of risk they have not assessed, through software nobody approved, producing output nobody verified.

The uncomfortable part is that this is not a training gap in the soft sense. Since February 2025, it is a legal obligation.


Confidence is not evidence

The single most important property of a large language model is that it does not look facts up. It predicts the next most likely token based on patterns learned from an enormous volume of text. That means it can produce a fluent, well-structured, entirely confident answer that is completely false — and it is not lying, because it has no mechanism for distinguishing a true sentence from a plausible-sounding one.

Fluency in tone is not evidence of accuracy. It is evidence that the model is good at language.

Nor is this a rare glitch awaiting a patch. Independent benchmarks are consistent about it. The HALoGEN study presented at ACL 2025 tested 14 leading models across roughly 150,000 generations and found hallucination rates reaching 86% in some subject areas. The DefAn benchmark measured factual hallucination between 59% and 82% across six major models. These are structural consequences of how the systems work, not bugs the next release removes.


What it costs when nobody checks

In 2023, lawyers in Mata v. Avianca filed a brief in the Southern District of New York containing six case citations produced by ChatGPT. Every one was fabricated — complete with plausible case names, courts and dates. The court imposed a $5,000 sanction.

What makes the case instructive is not that the model failed. It is that the model gave no signal that it had failed. The output looked exactly like real legal research. And a trained professional would have caught all six with a single verification step.

There is a second, independent failure mode: the training data itself can be wrong. Amazon built an internal recruiting tool trained on years of its own hiring history. That history reflected a bias toward male candidates, so the model learned to systematically downgrade CVs containing the word "women's". Amazon scrapped the tool. Biased, incomplete or outdated data does not announce itself — the model reproduces the defect silently, at scale, with full confidence.


The damage happens after deployment

The AI Incident Database, a public repository of real-world AI failures maintained by the Responsible AI Collaborative, has catalogued well over a thousand documented incidents — and its own analyses find that most harms surface only once a system is in production use, not during development or testing.

Read that pattern and the conclusion is hard to avoid: model risk becomes organisational harm through how people use the model. Which puts human behaviour, oversight and verification inside any serious AI risk programme — and puts employee training precisely where exposure can be reduced.


Articles 4 and 26

Regulation (EU) 2024/1689 — the EU AI Act — creates two obligations that bear directly on this.

Article 4 requires providers and deployers to ensure a sufficient level of AI literacy among staff dealing with the operation and use of AI systems. It has been in application since February 2025. Note the word deployers: this is not an obligation that lands only on the companies building models.

Article 26 requires that human oversight of high-risk AI systems be assigned to people with the necessary competence, training and authority. Several employment and education use cases fall into the high-risk category. Related deployer obligations carry penalties of up to €15 million or 3% of global turnover.

And regulators do not necessarily wait for demonstrated harm. On 30 March 2023, Italy's Garante imposed an immediate provisional restriction on the processing of Italians' personal data through ChatGPT. OpenAI suspended access in Italy and was given 20 days to report corrective measures; service resumed on 28 April after remediation. The measure was preventive. "Nothing has gone wrong yet" is not a defence.


Shadow AI closes the loop

Here is the trap. Employees use AI tools without approval or oversight — shadow AI — and the sequence writes itself. An unapproved tool produces unverified output. That output lands in a customer reply or a compliance document. The moment it leaves the building, the organisation is accountable, whether or not it ever approved the tool.

"We did not know what our employees were doing" is not a defence under the European framework. It is evidence of a governance failure.


Four controls that work

None of them are complicated, and none of them require new software.

Human verification before any AI output enters a decision or leaves the organisation.

No source, no decision. If a citation cannot be verified, it does not count. This one rule would have caught Mata v. Avianca in its entirety.

Documented governance defining what AI may and may not produce autonomously — documented, because documentation is what you will be asked for as evidence.

Training that makes scepticism habitual, so every employee knows that fluency is not accuracy and knows how to check.


Where this meets the rest of your compliance work

If your organisation is already in scope for NIS2, this pattern will be familiar: the obligation is not to have a policy, it is to demonstrate that a control is operating. AI literacy is no different. Attendance records, policy acknowledgements and defined oversight roles are the evidence — and, like every other control, they have to exist before someone asks for them rather than being assembled in a fortnight of panic. We wrote about that shift in why NIS2 compliance is no longer an audit moment.

The question is not whether your employees are using AI. They are, right now, with or without approval. The question is whether they have been trained to use it safely.


Euraika GRC tracks control evidence — including training and oversight assignments — continuously rather than at audit time. See how it works, or check your NIS2 position with the self-assessment.

Educational content, not legal advice. Key sources: Regulation (EU) 2024/1689, Articles 4 and 26; HALoGEN, ACL 2025; DefAn, arXiv:2406.09155; AI Incident Database (incidentdatabase.ai); Mata v. Avianca, S.D.N.Y., 2023; Italian Garante, measures of 30 March and 28 April 2023.