Why AI Incidents Are Different from IT Incidents
Traditional IT incident response is built around availability and integrity failures: a server goes down, a breach occurs, a system produces corrupted data. The failure mode is typically hard — something stops working — and the response is restoration or containment. That model doesn't map well to most AI system failures, which are soft, gradual, and often invisible to standard monitoring.
Silent degradation. A model trained on historical data will drift as underlying patterns change. An unemployment eligibility model trained before a major labor market shift may produce systematically wrong determinations without triggering any alert, because the system is technically operating. Nothing in the monitoring infrastructure flags an accuracy decline until outcome data accumulates — often months later.
Systematic directional errors. A model might be consistently wrong in a specific direction: over-approving fraud claims that match a particular profile, consistently underscoring applicants with non-standard document types, or applying a stale regulatory threshold that has since changed. Identifying the pattern requires outcome analysis, not system status monitoring.
Disparate impact on protected classes. A model that is accurate in aggregate may produce discriminatory outcomes across demographic subgroups. This requires disaggregated outcome monitoring — tracking accuracy by relevant population dimensions — not just overall uptime and error rates. Most state AI monitoring setups don't do this.
Data pipeline corruption. If an upstream data source changes its format, encoding, or coverage, model inputs can become meaningfully different from training data without any visible error. The model keeps running; its outputs silently degrade. This is particularly common in state benefits systems where federal data feeds change on program update cycles.
Effective gaming by bad actors. In fraud detection contexts, adversaries who observe decision patterns can adapt inputs to exploit decision boundaries. This looks like a system doing its job while the underlying effectiveness erodes.
Each of these failure modes requires different detection methods, different evidence standards for causation, different remediation strategies, and different communication obligations to citizens and oversight bodies. They don't fit into a standard major incident runbook.
The MANAGE Function and What It Actually Requires
The NIST AI Risk Management Framework (NIST AI 100-1, January 2023) organizes AI risk management into four functions: GOVERN, MAP, MEASURE, and MANAGE. Most state AI governance frameworks that adopt the AI RMF — and the framework is voluntary for state agencies, though widely referenced in state policies — invest heavily in GOVERN (policies, roles, accountability structures) and MAP (system inventories and risk categorization). MEASURE gets partial attention through pre-deployment testing and procurement assessments.
MANAGE is where ongoing operational responsibility lives. It covers risk prioritization and response based on monitoring results, implementation of strategies to maximize benefits and minimize negative impacts throughout the system lifecycle, management of third-party AI risk, and documented recovery and treatment protocols. These aren't aspirational — they're operational capabilities that require staff, procedures, tooling, and clear decision authority.
A state that issues an AI executive order, builds a use-case inventory, and requires vendor compliance certifications has addressed GOVERN and parts of MAP. It has not addressed MANAGE unless it has people who can detect a model drift event, authority structures that allow them to act, and documented procedures for what happens when they do.
The enforcement gap in state AI policy reinforces this. A 2026 analysis of state government AI rules by Smarsh found that fewer than half of jurisdictions with AI policies actively enforce them, and only nine specify penalties. Enforcement gaps are often a symptom of the same problem: policy that describes expected outcomes without operational mechanisms for monitoring them.
Classifying State AI Incidents
The operational gap starts with definition. Most state AI policies don't define what an "AI incident" is, which means there's no trigger for response and no clear ownership for investigation. A working classification organizes by consequence, not cause:
Tier 1 — Immediate response. Failures affecting safety, liberty, or access to essential services: a benefits system denying claims in a pattern that suggests disparate impact on a protected class, a child welfare tool producing systematically incorrect removal recommendations, a fraud detection engine misclassifying a protected characteristic as a risk proxy. Response is immediate human review override, system pause if the failure cannot be isolated to a specific input category, and formal incident documentation within 24 hours. These systems include anything that informs or produces decisions about eligibility for essential programs, child placements, or criminal justice proceedings.
Tier 2 — Time-bounded investigation. Failures affecting service accuracy or program integrity where citizens are harmed but the failure is not acute: an eligibility model whose approval rate has shifted materially without a corresponding policy change, a procurement scoring system producing unexplained variance from prior benchmarks, a workforce screening tool that produces anomalous outcomes for a specific document type. Investigation within five business days, with interim human review for decisions in the affected category during the investigation period.
Tier 3 — Documented drift. Gradual accuracy degradation within tolerable thresholds but outside the monitoring bounds established at deployment. Requires root cause analysis and a remediation plan within 30 days, vendor notification if the model is vendor-operated, and updated performance baselines. No immediate operational pause required, but documented and tracked.
Every AI system in a state's production inventory should have a tier assignment based on its use case, documented monitoring baselines established at deployment, and a clearly named incident owner. Without an owner, incidents get discovered during post-hoc audits rather than active monitoring — and the delay between failure and detection is where most of the citizen harm accumulates.
The Vendor Dependency Problem
The practical complication is that most states don't own or control their AI models. Vendor-operated systems create response problems that governance frameworks routinely miss:
Visibility gaps. The state may have no direct access to model performance metrics, training data versions, or inference logs unless the vendor contractually provides them. Contracts that describe accuracy at procurement but don't require ongoing monitoring transparency leave the state operationally blind to model changes.
Rollback authority. If a vendor pushes an unannounced model update — which happens routinely in SaaS-delivered AI products — the state's ability to roll back to a prior version depends entirely on whether the contract specifies that right. Most legacy state AI contracts don't.
Response timeline mismatches. A state's incident response procedure may require a 24-hour Tier 1 response. A vendor SLA may promise different terms. The gap becomes a governance failure in an actual incident, especially when the vendor is the only party with access to root cause information.
Root cause access. Investigating a disparate impact failure requires access to the model's decision logic at the input level. Black-box models operated under commercial licenses may contractually bar that access — and even when they don't, vendors have limited incentive to facilitate it quickly.
These gaps are straightforward to address at the contracting stage: require monitoring log access, define rollback rights explicitly, establish incident notification timelines, and specify what root cause investigation access the state retains. They are significantly harder to negotiate retroactively, especially under an active incident.
The state AI policies issued in 2026 by Arkansas and New York both extend AI governance obligations to vendors and third parties — Arkansas's February 2026 policy covers "all Executive Branch employees, contractors, vendors, and third parties who design, procure, implement, or use AI tools." Extending that coverage to production monitoring requirements, not just initial use restrictions, is the next operational step most states haven't taken.
Building the Capability Before the Incident
Constructing an AI incident response capability is a three-part operational problem.
Monitoring baselines for every production system. At deployment, document the system's expected performance: overall accuracy by output category, performance disaggregated by relevant population dimensions, approval and denial rate stability within defined bounds, and data pipeline health indicators. The baseline is the threshold that triggers escalation. Without one, drift has no reference point and Tier 2 incidents are invisible until they become Tier 1.
An ownership map. For each Tier 1 and Tier 2 system, identify in writing: who calls the incident (incident owner), who investigates the technical cause (data or IT team), who holds remediation decision authority (agency IT director or CISO), and who handles external communication to affected citizens and oversight bodies. This map needs to exist before an incident, not during one. The most consistent finding from post-incident reviews is that the authority was undefined — everyone assumed someone else owned the response.
A tabletop exercise. Use a specific production system. Introduce a plausible failure scenario — demographic accuracy divergence detected in quarterly outcome review, or an upstream data feed change that went unnoticed for 60 days. Walk the team through detection, tier classification, investigation initiation, vendor engagement, and communication obligations under actual response timeline pressure. Tabletop exercises surface authority gaps, vendor notification delays, and unclear tier boundaries that policy documentation obscures. They also establish that the monitoring infrastructure can actually detect the failure mode in question — a finding that frequently turns out to be false until tested.
States that have done this work report a consistent finding: the policy exists, the inventory exists, but nobody owns the pager when it rings.
Sources and further reading
- NIST AI Risk Management Framework (AI 100-1), National Institute of Standards and Technology, January 26, 2023: https://doi.org/10.6028/NIST.AI.100-1
- NIST AI RMF Playbook (Manage function action examples), NIST Trustworthy and Responsible AI Resource Center: https://airc.nist.gov/Docs/2
- Arkansas Executive Branch AI Acceptable Use Policy, effective February 26, 2026: https://sas.arkansas.gov/wp-content/uploads/Policy_AI-Acceptable-Use.pdf
- New York State AI Acceptable Use Policy NYS-P24-01, updated June 24, 2026: https://its.ny.gov/acceptable-use-artificial-intelligence-technologies
- Smarsh, "How U.S. States Regulate Government Use of AI" (analysis of state AI enforcement): https://www.smarsh.com/blog/thought-leadership/state-ai-regulation-government-use
- CISA, "Guidelines for Secure AI System Development" (joint guidance, November 2023): https://www.cisa.gov/resources-tools/resources/guidelines-secure-ai-system-development
Spartan X's Arbiter capability directly addresses the continuous verification and monitoring gap that state CISOs face in production AI deployments — building the assurance mechanisms that let agencies detect when a model is no longer operating within its defined performance envelope and reconstruct what happened when it wasn't. For state agencies operating AI in high-consequence programs, the same discipline that makes AI verification defensible in operationally demanding environments applies to the monitoring baselines, incident classification, and vendor accountability structures this article describes.
