ISO Compliance Insights & Best Practices
CART
ISO Compliance Insights & Best Practices
AI data quality risk is the chance that flaws in the data used to train, test or run an AI system lead to wrong, unfair or unsafe results. Models learn from data, so unrepresentative, mislabelled, outdated or unlawfully obtained data becomes a defect built into the system. Many AI failures blamed on the algorithm are, on inspection, data problems that could have been found earlier.
This guide explains the main dimensions of AI data quality risk, how to assess them, which controls reduce the risk, how to monitor data in operation and what evidence to keep. It is general guidance that you should adapt to your systems and to the legal requirements that apply.
The saying “garbage in, garbage out” is truer for AI than for most systems. A model finds patterns in whatever it is given, including errors, biases and artefacts. It then applies those patterns at scale, with an appearance of objectivity that makes the mistakes hard to challenge.
Regulators and standards recognise this. The EU AI Act sets data governance requirements for high-risk systems, including that training, validation and testing data be relevant, sufficiently representative and, to the best extent possible, free of errors. The NIST AI Risk Management Framework, described in the NIST AI Risk Management Framework, treats data quality and provenance as part of mapping and measuring risk. Assess AI data quality risk explicitly, and record it in the AI risk register.
Free AI risk assessment
Which of your AI systems could harm people, or you?
List your AI systems, models and data, pick from 38 AI risk scenarios, rate them for the people affected and for you, and plan treatment with ISO 42001 Annex A controls. You get a heat map, a process score and the findings an auditor would raise, free.
Run the free AI risk assessment → or View premium report sample
Ask whether the training data reflects the population and situations in which the system will be used. A model trained on data from one region, age group or device type may perform poorly elsewhere. A medical model trained mostly on one demographic may miss conditions in others.
Compare the data distribution with the target population, and test performance across subgroups. Use the results from AI bias testing to decide whether more data, reweighting or a narrower use case is needed. Record any known gaps and limits on use, and make them visible to users.
Supervised models learn from labels, and labels are often produced by people or by proxies. Human labellers make mistakes, disagree and bring their own assumptions. Proxy labels, such as using past decisions as the “correct” answer, can encode historical bias.
Check label quality: sample and review labels, measure agreement between labellers, document guidelines and audit disputed cases. Ask whether the label really measures what you want. For example, “arrests” is not the same as “crime”. Where labels are weak, consider whether the model should be used for high-stakes decisions.
| Dimension | Question | Example risk |
|---|---|---|
| Representativeness | Does the data reflect the people and conditions the system will meet? | Model performs poorly for under-represented groups |
| Accuracy and labelling | Are values and labels correct and consistent? | Noisy labels teach the model wrong patterns |
| Completeness | Is important data missing, and is missingness systematic? | Missing fields correlate with a protected group |
| Provenance and lawfulness | Where did the data come from and may we use it? | Scraped or unlicensed data creates legal exposure |
| Timeliness and drift | Is the data current, and does it stay similar over time? | Behaviour changes and the model degrades |
Missing data is rarely random. Fields may be missing more often for certain groups, channels or time periods, and the way missing values are handled can bias results. Analyse which fields are missing, why and for whom, and test the effect of different handling methods.
Check for gaps in time and coverage too. A dataset that stops before a major change in behaviour, such as a policy change or a market shock, will not prepare the model for the new conditions. Note such limits and plan more frequent retraining or monitoring.
Know where every dataset came from, who owns it, what rights and licences apply and whether personal data was collected lawfully and for a compatible purpose. Unclear provenance creates legal, ethical and reputational risk, and it can force a model to be withdrawn.
Keep a data inventory or datasheet for each dataset: source, collection method, date, licence, personal data content, known limitations and approvals. Where personal data is used, link to your assessments; see DPIAs for AI systems and ROPA purposes and lawful bases. For third-party datasets and models, ask suppliers for documentation and warranties.
Data can also be attacked. Poisoning attacks insert malicious or misleading examples into training data to manipulate the model, and leaks can expose sensitive records. Protect data pipelines with access control, integrity checks, logging and separation of environments.
Verify that data used in testing is independent from training data, since leakage between the two inflates performance estimates. Keep versions of datasets so a model can be reproduced and, if needed, retrained after a problem is found.
Once deployed, the data a system sees will change. Customer behaviour shifts, sensors degrade, upstream systems change formats and the world moves on. Monitor input distributions, missing rates, out-of-range values and performance against ground truth. See AI model drift for methods.
Set thresholds and alerts, and define what happens when they trigger: investigation, fallback, retraining or pause. Connect alerts to AI incident management, so serious data problems are handled with the right urgency.
Effective controls include data standards and checklists, profiling and validation at ingestion, labelling guidelines and audits, representativeness analysis, documented provenance and approvals, security controls, monitoring and periodic review. Assign a data owner for each dataset who is accountable for its quality.
Build the controls into the development lifecycle rather than adding them at the end. A gate that requires data documentation and quality checks before training saves rework and provides evidence that the checks happened. Record residual risks and link them to your AI risk treatment plan.
Frequent errors include assuming more data means better data, using convenient data instead of representative data, ignoring label quality, failing to document provenance, testing on data that leaked from training, skipping monitoring after deployment and leaving no owner for the dataset. Another is treating data quality as purely technical, forgetting legal and ethical dimensions.
Avoid these by combining statistical checks with human review, documenting datasets, involving legal and privacy early and making data owners accountable.
Synthetic data can fill gaps and protect privacy, but it inherits the biases of the data it was generated from and may miss rare real-world cases. Validate it against real data, and label it clearly in documentation. For purchased or licensed datasets, verify the supplier’s collection methods, check for embedded personal data and confirm that the licence covers your use. Treat gaps in supplier information as risks to record and to raise in due diligence.
A retailer builds a model to forecast demand for stores. Data checks show that two regions have far fewer records, that promotions were logged inconsistently and that a supplier feed changed units halfway through the period. The team fixes the feed, adds a promotion field standard, collects more regional data and reports the remaining coverage gap.
After launch, monitoring flags that store openings change patterns, and the team schedules retraining. The dataset documentation, quality checks and monitoring results are stored with the model record. The risk is rated medium and reviewed quarterly.
Keep a short data record for each model: sources, dates, rights, personal data content, quality checks performed and their results, known limitations and approvals. Link it to the model inventory entry and to the risk assessment. When an auditor, regulator or customer asks how you know the data is fit for purpose, you can show the evidence rather than reassurance.
Review the record whenever data sources or use cases change, and note who reviewed it. Good documentation also helps new team members understand the model and reduces the chance that a future change unknowingly breaks an assumption.
If you want a report and workbook that connect data risks, controls, monitoring and treatments in one place, the AI Risk Assessment Report and Workbook provides a structured layout for AI risk assessment. Whatever the tool, managing AI data quality risk means knowing your data, testing it against the real population and watching it after launch.
The risk that flaws in the data used to train, test or run an AI system, such as bias, errors, gaps or unlawful sourcing, lead to wrong, unfair or unsafe results.
No. Large but unrepresentative or mislabelled data can make a model worse or unfair. Quality, coverage and provenance matter as much as volume.
Sample and review labels, measure agreement between labellers, document guidelines and question whether the label measures the outcome you care about.
Yes. Input data changes over time, so monitor distributions, missing values and performance, and define actions when thresholds are crossed.
A named data owner for each dataset, working with the model owner, privacy, security and legal teams.
September 30, 2026
