ISO Compliance Insights & Best Practices
CART
ISO Compliance Insights & Best Practices
AI incident management is the discipline of noticing when an AI system has caused, or nearly caused, harm, and responding quickly and honestly. Traditional incident processes cover outages and breaches, but AI failures can be quieter: a model drifts and treats a group unfairly for months, a chatbot gives dangerous advice, a fraud model blocks thousands of legitimate customers. If your process cannot recognise these events, they will not be handled.
This guide explains how to define AI incidents, how to detect and triage them, how to contain harm, when reporting duties may arise, how to find root causes and how to learn from them. It is general guidance, not legal advice, and reporting duties depend on the system, sector and jurisdiction.
AI systems fail differently from ordinary software. A conventional program either works or crashes. A model can keep running while producing worse and worse results, harming people without triggering an alert. Harms may be spread thinly across many users, so no single complaint looks serious, and the cause may lie in data rather than code.
That is why AI incident management needs specific definitions, detection methods and playbooks. It should plug into your general incident process, so people know one route for raising problems, but add the questions and skills that AI cases demand: what did the model do, for whom, since when and why?
Write a clear definition that includes actual harm and credible near misses. An AI incident might be an event in which the system produced outputs or decisions that harmed or could harm people, violated law or policy, breached fundamental rights, caused significant financial loss or seriously damaged trust. Include misuse and attacks.
Provide examples in the policy and use severity levels to guide response. The EU AI Act, in the EU AI Act, including its provisions on serious incident reporting, includes duties for providers and deployers of high-risk systems regarding serious incidents, so check whether your systems fall within its scope and what triggers a report.
Combine several sources. Technical monitoring can flag drift, error spikes, unusual output patterns and fairness metric changes, as discussed in AI model drift. People can raise concerns: staff, reviewers, customers and suppliers. Complaints, appeals and override patterns are valuable signals.
Make reporting easy. A single channel and a short form, with no blame for good-faith reports, will surface more problems. Train front-line staff and reviewers on what to look for, and connect the human oversight of AI process, since reviewers often see failures first.
| Incident type | Example | First response |
|---|---|---|
| Harmful or unsafe output | Chatbot gives dangerous medical advice | Suspend feature, preserve logs, notify owner |
| Discriminatory outcome | Model rejects one group at a much higher rate | Pause automated decisions, review affected cases |
| Data leakage | Model reveals personal data in responses | Contain, assess breach duties, fix prompts or data |
| Performance failure or drift | Accuracy falls sharply after a data change | Switch to fallback, investigate data pipeline |
| Misuse or attack | Prompt injection or adversarial manipulation | Block, investigate, strengthen guardrails |
When a report arrives, assess it quickly: what happened, which system and version, who is affected, how many, how severe and whether harm is continuing. Assign a severity level and an incident owner, and set target times for response and update.
Involve the right people early: the system owner, data science, security, privacy, legal, communications and, where needed, senior management. Use a checklist so nothing is missed, such as whether personal data is involved and whether a regulator or customer must be notified.
The first goal is to stop the harm. Options include pausing the system, switching to a fallback or manual process, rolling back to a previous model version, disabling a feature, tightening guardrails or blocking a malicious user. Choose the least disruptive action that works and record the decision and time.
Then deal with people already affected. Identify them, review their cases, correct decisions and offer remedies where appropriate. For discriminatory outcomes, this may mean re-running decisions. Keep evidence: logs, inputs, outputs, model versions and configuration, since you will need them for root cause analysis.
Decide who must be told. Depending on the case, that may include the affected people, customers, regulators, data protection authorities, sector supervisors or insurers. If personal data was involved, personal data breach rules may apply, with tight deadlines. For high-risk AI systems, the AI Act sets rules on reporting serious incidents to market surveillance authorities, so confirm the requirements for your role.
Prepare templates and decision criteria in advance. Make sure legal and privacy review notifications. Keep a log of who was told what and when. Honest, timely communication usually protects trust better than delay.
Once the situation is stable, investigate why it happened. Look at data (quality, coverage, shifts), model (design, training, testing), deployment (configuration, integration, prompts), oversight (review, monitoring) and organization (incentives, pressure, unclear ownership). Use structured methods such as five whys or fault trees.
Avoid stopping at “the model was wrong”. Ask why testing did not catch it, why monitoring did not alert and why oversight did not intervene. The answers point to fixes that prevent similar incidents, not just this one.
Every incident should produce actions: fix the model or data, add tests, improve monitoring, update risk assessments, retrain reviewers, change contracts and revise policies. Assign owners and dates and track closure. Update the AI risk register and the AI risk treatment plan with new or changed risks.
Free AI risk assessment
Which of your AI systems could harm people, or you?
List your AI systems, models and data, pick from 38 AI risk scenarios, rate them for the people affected and for you, and plan treatment with ISO 42001 Annex A controls. You get a heat map, a process score and the findings an auditor would raise, free.
Run the free AI risk assessment → or View premium report sample
Share lessons across teams in a blameless way. Track metrics such as time to detect, time to contain, number of near misses reported and repeat incidents. A rising number of near-miss reports is often a good sign: it means people trust the process.
Report incident trends to the governance forum, and include serious incidents in board reporting. Where a supplier provides the model, agree in the contract how incidents will be notified, what logs you will receive and how fixes will be delivered. See third-party AI impact assessment for supplier questions.
Record incidents in your AI model inventory against each system, so its history is visible to anyone assessing it.
Frequent errors include treating AI failures as ordinary bugs, having no definition of an AI incident, relying only on customer complaints for detection, keeping no logs, blaming individuals, failing to check affected populations, delaying notification and never updating the risk assessment afterwards. Another is having no fallback, so pausing the system means stopping the business.
Avoid these by defining incidents, monitoring proactively, preserving evidence, preparing fallbacks and running exercises for likely scenarios.
A bank’s monitoring detects that a fraud model now blocks a much higher share of transactions from a particular region after a data update. The incident is triaged as high severity. The team switches those decisions to manual review, preserves logs, identifies about two thousand affected customers and reverses wrongful blocks.
Root cause analysis finds that a new data feed encoded location differently, and testing had not covered that field. The bank fixes the pipeline, adds tests and a fairness monitor, updates its risk register and reports the event to management. Customers receive an explanation and an apology.
Write short playbooks for the most likely incident types and run tabletop exercises at least once a year. Include technical, legal, privacy and communications staff, and test decisions such as when to pause the system. Record what worked and what did not, and update the playbooks.
Check that contacts are current, logs are actually being kept and rollback is technically possible. An exercise that reveals you cannot roll back a model is much cheaper than a real incident that does. Bring suppliers into the exercises where they run critical models.
If you want a report and workbook that connect AI risks, controls, monitoring and incident response actions, the AI Risk Assessment Report and Workbook provides a structured layout for AI risk assessment. Whatever the tool, effective AI incident management defines what counts as an incident, detects it early, contains harm, reports honestly and turns each event into improvement.
An event where an AI system caused or could have caused harm, breached law or policy, or seriously damaged trust, including misuse and credible near misses.
AI failures can be gradual, spread across many people and caused by data rather than code, so they need specific detection, triage questions and root cause methods.
It depends on the system, your role and the jurisdiction. High-risk systems under the EU AI Act carry serious incident reporting duties, and personal data breach rules may also apply.
Logs, inputs, outputs, model version, configuration, decisions taken and communications, so the cause can be investigated and actions justified.
Yes. They reveal weaknesses before harm occurs and a healthy reporting culture depends on treating them as learning opportunities.
September 30, 2026
