← All posts

ai:incidents

When an AI system gets it wrong in production: an incident plan.

AI incident response is the plan you write before an AI system in production goes wrong: what counts as an incident, who declares one, how to contain it, who gets told, and how the failing case becomes a test. It covers the organization's response; the system's own controls, such as permissions and a stop switch, cover only the system.

This guide is for the operations and risk owners of a chatbot that answers customers, an assistant that drafts replies, or an agent that acts in company systems. Each section is a decision to make and write down before the first incident.

What an AI incident response plan covers.

The plan is short and written in advance: the incident types the team recognizes, severity levels with an owner for each, the containment moves available, who is told and in what order, how the cause is investigated, and how a fix is gated before it ships. Keep it to one page per system, so the person on call can use it during an incident.

The structure comes from security practice: NIST's incident response guidance for cybersecurity (SP 800-61 Rev. 3, a CSF 2.0 community profile) assists organizations with incorporating incident response recommendations throughout their cybersecurity risk management activities. The AI Risk Management Framework (AI RMF 1.0) puts the same idea in AI terms: its Manage function says risk treatment comprises plans to respond to, recover from, and communicate about incidents or events, and subcategory MANAGE 4.1 calls for post-deployment monitoring plans that include incident response, recovery and change management. Your plan applies that guidance to one system.

What counts as an AI incident.

The wider picture is tracked publicly: the OECD's AI Incidents and Hazards Monitor (AIM) describes itself as an automated media discourse monitor that documents AI incidents and hazards to give policymakers, practitioners and stakeholders insight into their risks and harms.

Write your own list down. Five types to start from:

  • Wrong or harmful answers. The system misstates a price, a policy or a right, confidently, to a customer, or produces content that harms someone.
  • Data exposure. The system retrieves or reveals data its user was never meant to see, or sends content to the wrong recipient.
  • Actions taken in error. An agent with write access to company systems changes, deletes or sends something it should have left alone.
  • Runaway cost. A loop or a mis-scoped task spends past every expectation, with no ceiling in place.
  • A model change that breaks a workflow. The provider swaps or updates a model and the output quality, format or language support shifts under a live process.

Before it happens: owner, severity levels, switch-off, fallback.

Four things exist before the first incident. A named owner per system, responsible for the plan and the follow-through. Severity levels written when calm, from a wrong answer one customer saw to data exposure across the tenant, each naming who is awake and who decides. A switch-off that works and has been tested: where it lives, who can pull it, and what pulling it takes down. And a manual fallback: how the work ran before the AI step, documented so the team can step back to it without inventing the process mid-incident.

For agents that act in systems, the system's own controls are the first line, and our guide to AI agents in production covers them: scoped permissions per tool, action and spend caps, approval for irreversible steps, an action log and a stop switch. The incident plan sits above those controls and answers for the day they are not enough.

Contain: switch off or narrow, then preserve the evidence.

Containment is a decision with options, taken by the owner the severity level names: switch the whole feature off, narrow it (disable one flow, one channel, one audience, cap the spend), or leave it live with a human now approving every output. Full switch-off stops everything the feature does; narrowing keeps the rest of the service running while the failing part is off.

Preservation comes immediately after. Freeze the request logs, the prompts and retrieved context, the exact model version and the outputs as sent, before retention windows rotate or someone edits the prompt mid-investigation. The cause analysis and any notification rest on this evidence.

Tell people: users, audit, and the data protection duties.

The plan lists who is told, in what order. Affected users first where the incident touched their data or the answers they received; internal audit and the data protection lead close behind; and counsel, who reads the legal duties.

Where UAE federal law applies, the text to read is Article 9 of Federal Decree-Law No. 45 of 2021 (PDPL), Reporting Personal Data Breach: the Controller notifies the Bureau of a breach or violation of personal data that would prejudice the privacy, confidentiality and security of data, within the period and in the manner the Executive Regulations set, accompanied by, among other items, a description of the breach, its expected effects and the corrective measures taken or suggested; and in all cases the Controller notifies the Data Subject where the breach prejudices the privacy and confidentiality of the security of their personal data. We describe the text; whether and when it binds your entity is your counsel's call.

The AI RMF describes the same step as an outcome: MANAGE 4.3 says incidents and errors are communicated to relevant AI actors, including affected communities, and that processes for tracking, responding to, and recovering from incidents and errors are followed and documented.

Find the cause: prompt, data, model version, integration.

Four places hold the cause; check them in this order. The prompt: a wording that steers the model wrong on inputs you never tested. The data: a retrieval index serving stale or mis-scoped documents, so the system repeated sources that were wrong. The model version: an update that changed behavior on your workload. The integration: the plumbing around the model, where fields are dropped, truncated or mapped badly.

Where the cause is a missing review step, the fix is structural: our guide to human-in-the-loop AI workflows covers which outputs a person reviews, who reviews them and how corrections are used. For errors that must not reach a customer, add that review step as well as fixing the prompt.

Fix and prevent: the failing case joins the release tests.

The incident closes when the failing case is a test. Add the input, the wrong output and the expected output to the suite that gates releases, so a release that repeats the failure fails the suite. Run that suite against model updates, since a fix that holds on today's model can slip on the next one. Our AI production readiness checklist covers what a go-live review checks; after an incident, find the item on it that would have caught the failure.

Between incidents, someone has to watch the system. For the AI systems out:grow builds, managed AI operations does that: watching for failures, updating as models change, and fixing prompts before a broken run reaches your team or your users.

Do it yourself, or bring us in.

Everything above can be written in-house in a working session per system: the incident list, the severity levels with owners, the tested switch-off, the fallback, the notification order. Rehearse it once, fix what the rehearsal breaks, and date the page.

Outside help fits when you want the engineers who build the system to keep it running after launch, monitored and updated as models change.

Book a 30-minute call

Not ready for a call? Take the 10-question AI readiness check

Frequently asked questions.

Is a wrong chatbot answer an incident?

It is when it crosses the line your severity levels drew. One wrong answer caught and corrected by support is a defect; the same wrong answer given to every customer who asks about a refund, in words that misstate their rights, is an incident. Decide by the effect and the reach, and write the line down before the case arrives.

Who is liable for what a chatbot told a customer?

One public decision gives a fact pattern. In Moffatt v. Air Canada (2024 BCCRT 149), the Civil Resolution Tribunal records that the airline's website chatbot suggested a customer could apply for bereavement fares retroactively, that Air Canada staff later said that was not permitted, and that the airline argued it could not be held liable for the chatbot's information. The tribunal mostly allowed the customer's claim and ordered the airline to pay CAD 812.02, including CAD 650.88 in damages. Other forums can decide differently, and the reading for your systems stays with counsel.

Should we tell customers?

When the incident touched their data or the answers they received, decide it in the plan in advance: who tells them, through which channel, in what words. Where UAE federal law applies, Article 9(2) of the PDPL says the Controller shall notify the Data Subject where the breach would prejudice the privacy and confidentiality of the security of their personal data. Counsel reads what that means for your entity.

How fast should we be able to switch off?

As fast as your preparation allows, and you learn your number only by rehearsing. The switch lives where the on-call can reach it without a release cycle, and a rehearsal measures what switching off costs the business.

Keep reading.

:

Running AI agents in production: permissions, limits and rollback.

:

Human-in-the-loop AI workflows: running the review step in production.

:

AI production readiness checklist: what to prove before go-live.