The documentation problem is a governance problem
AI is embedded in state program administration at a depth most state CIOs cannot fully account for. Eligibility platforms flag applications for manual review based on model outputs that arrived as a vendor feature update, not a deliberate procurement decision. Fraud detection tools score claims before a human examiner sees them. Document processing pipelines classify intake submissions and route them to caseworkers. Chatbot triage handles the first tier of call center traffic.
Most of these deployments happened through agency-level procurement, often as add-on capabilities within enterprise vendor contracts. They were evaluated as product features, not AI systems. The result is that many state agencies are now running consequential, model-driven logic against program populations without being able to answer the basic governance questions: what systems are running, what decisions they influence, how they were validated, and who is accountable when one produces a wrong output.
This is not a technology problem. The systems are running. It is a documentation and governance problem — and that gap is becoming an audit exposure.
What federal oversight frameworks are beginning to expect
OMB Memorandum M-25-21, issued April 3, 2025, requires federal executive agencies to designate a Chief AI Officer, maintain an AI governance board, track and report their AI use cases, and apply specific minimum practices — including pre-deployment testing, ongoing monitoring, and human oversight — for high-impact AI systems. Federal agencies are not state agencies. M-25-21 does not directly bind a state Department of Health or a state labor agency.
But the federal agencies that oversee major joint federal-state programs — the Centers for Medicare and Medicaid Services for Medicaid and CHIP, the Food and Nutrition Service for SNAP, the Department of Labor for unemployment insurance — are implementing M-25-21 internally. When those agencies conduct program reviews of state-administered programs, the documentation practices they apply to their own systems shape the questions they ask about yours.
The compliance mechanism is not primarily regulatory. It is practical and operational:
- Single audit requirements. Federal grant recipients spending $1,000,000 or more in federal awards in a fiscal year must undergo a single audit under OMB's Uniform Guidance (2 CFR Part 200, §200.501; the threshold was raised from $750,000 to $1,000,000 for fiscal years beginning on or after October 1, 2024). Auditors examine internal controls over federally-funded programs — and IT systems that administer those programs fall within that scope. AI components embedded in program systems represent a documented internal control question for auditors assessing whether controls are designed and operating effectively.
- Program review walkthroughs. Federal program reviews typically include systems walkthroughs for eligibility and case management platforms. A reviewer asking what automated tools affect eligibility determination is asking about AI. If the agency cannot answer specifically, the reviewer notes an information gap — one that generates follow-up requests and can escalate into a formal finding.
- Conditions of participation. Federal programs include documentation, security, and systems requirements as conditions of state participation. CMS's MARS-E framework (Minimum Acceptable Risk Standards for Exchanges), for example, requires documented security and privacy controls for state-based Health Insurance Exchange systems established under the ACA. As AI components become embedded in those systems, documentation expectations follow.
The state that cannot answer a reviewer's question about its AI systems does not necessarily fail an audit on that basis alone. But it generates findings, corrective action requests, and additional scrutiny for every claim it cannot support — and those findings carry into the next review cycle.
What the documentation gap actually looks like
The documentation gap is rarely that nothing exists. It is that what exists is fragmented, vendor-held, or scoped to the wrong question.
A typical state agency holds:
- Vendor contracts — but not documentation of what model versions run, against what training data, or when the vendor last updated the underlying model
- Change request and help desk logs — but not documentation of how system decision logic changed when the vendor pushed a model update
- Testing and acceptance records — from initial deployment, not from subsequent vendor updates
- Security review documentation — scoped to the system boundary and data flows, not to the AI component specifically
What a federal program reviewer or auditor needs differs in kind, not just in depth:
- What AI components are embedded in which program functions
- What the system produces — a score, a flag, a classification, a recommendation — and how that output connects to a human decision
- What population and time period the system was validated against, and what error rate targets were established
- What safeguards exist for high-risk outputs: human review thresholds, appeal pathways, override documentation
- Who at the agency owns accountability for each system and how changes are authorized
The gap between what a state typically holds and what a reviewer needs is what an AI use-case inventory closes.
What a useful AI inventory actually contains
The NIST AI Risk Management Framework, published January 26, 2023, defines four core functions — Govern, Map, Measure, Manage — that structure how organizations document and manage AI across its lifecycle. The Map function is the operational foundation: establishing what systems exist, what they do, and where risk concentrates, before attempting to measure or manage it. An AI use-case inventory is the institutional implementation of that Map.
For a state program office, a practical inventory entry contains:
System identification. System name, vendor, version, and deployment date. The program or function it operates within. A brief description of what the system does.
Decision scope. What outputs the system produces — score, classification, flag, recommendation. What human or automated decisions those outputs connect to. Which program population is affected and at what volume.
Data handling. What personally identifiable information categories the system processes. Where input data originates and where outputs flow. Data retention and deletion practices relevant to the AI component.
Validation basis. What population and time period the system was validated against. What performance targets — accuracy, false positive rate, false negative rate — were established at deployment. When the system was last assessed for performance drift against current program populations and conditions.
Accountability. Named technical owner inside the agency with authority over the system's configuration and operation. Named program owner with authority over whether and how the system is used. Escalation path for disputes about AI outputs.
Human oversight. What thresholds or conditions trigger human review of AI outputs. What the human review process requires of the reviewer. How overrides are documented.
Each entry should be completable by the agency from its own records — not from the vendor's assurances. Entries where fields cannot be filled from agency records identify specific documentation gaps that are operational risks independent of any audit.
Five scenarios to design the inventory against
Federal program review. A CMS, FNS, or DOL team reviews program administration and asks what automated tools are used in eligibility determination, fraud detection, or case management. The question is answerable in under an hour from a current inventory — or it triggers a multi-week scramble across vendor contracts and program staff.
Constituent complaint or advocacy inquiry. An applicant claims their benefits were incorrectly denied or delayed by an automated system. The agency needs to reconstruct what the system produced, when, and why — and demonstrate that a human review pathway existed and was available. The inventory identifies where to look and what the oversight process requires.
Vendor contract renewal. The agency evaluates whether an AI component of an enterprise system is performing as required and whether vendor model updates have materially changed the system's behavior. Without a validation baseline in the inventory, the agency has no independent basis for assessing the vendor's characterization of its own system.
Single audit internal controls finding. An auditor identifies a gap in how AI systems are governed in federally-funded program administration. The corrective action plan must name the affected systems, the specific gaps, and remediation steps with timelines. The inventory is the starting point for any coherent corrective action.
Legislative inquiry. A state legislative committee asks what AI systems the agency uses and what decisions they affect. An accurate, organized response — possible only from existing documentation — signals institutional competence. An inability to answer specifically signals something else.
Building the inventory: practical sequence
For most agencies, building a functional AI inventory is a three-to-six month effort at deliberate pace — longer if AI is embedded in vendor systems without contract provisions giving the agency access to technical documentation about those components.
1. Define scope. Establish what the agency means by "AI" for inventory purposes. A working definition for program administration contexts: any vendor or agency-built system that uses machine learning, statistical modeling, or rules derived from training data to produce outputs that influence program decisions or case routing. This definition includes deployed features inside enterprise platforms, not only standalone AI tools procured as such.
2. Issue a structured survey to each program area. Ask what AI-capable tools are in use, who holds the vendor contract, what program functions the tool affects, and who the technical and program contacts are. Budget two survey rounds — the first will undercount, because program staff do not routinely identify vendor product features as AI systems.
3. Review contracts for documentation provisions. Identify which contracts require the vendor to provide information about AI components — model update notifications, training data documentation, performance reporting, or audit access rights. Contracts without these provisions represent documentation gaps the agency cannot close without renegotiation. Flag them for the next renewal cycle.
4. Apply a risk tier to each system. Systems that affect individual eligibility determinations, fraud findings, or benefit amounts require more documentation depth than systems used for operational analytics or aggregate reporting. A three-tier classification — high (individual program determinations), medium (case routing and prioritization), low (internal operational) — focuses the inventory work on where accountability exposure is greatest.
5. Assign documented accountability. Name the technical and program owner for each system in writing, with a defined escalation path. Accountability that exists only informally does not satisfy a corrective action request and provides no operational clarity when a system behaves unexpectedly.
6. Establish a review and update cycle. The most common inventory failure is not the initial build — it is the first update cycle that does not happen. Require program areas to report material changes to AI system configurations or vendor model updates, and schedule an annual inventory review as a standing governance activity.
The decision the inventory enables
An AI inventory is not itself a safeguard. It does not validate that deployed systems are performing correctly or that human review processes are adequate. What it does is establish the documentation baseline without which no meaningful governance is possible.
A state that can name its AI systems, describe what they do, identify who owns them, and explain what oversight exists is in a fundamentally different position than one that cannot — with federal reviewers, with program populations who interact with those systems, and with state leadership making decisions about AI investment and risk.
The honest question a state CIO needs to answer is not "should we document our AI systems" but "what would we need to hand a federal reviewer today, and can we produce it?" Running that question against each major program area without prior preparation will identify the documentation work that is already overdue.
Spartan X has worked with state technology programs to establish AI governance frameworks — inventorying deployed systems, structuring documentation requirements, and defining accountability mechanisms that hold up under operational conditions and external review, not just in policy documents. The inventory is almost always where that governance work has to begin.
Sources and further reading
- Office of Management and Budget, M-25-21: Accelerating Federal Use of AI through Innovation, Governance and Public Trust (April 3, 2025) — current operative federal AI governance memo; requires CAIO designation, AI use-case tracking and reporting, and minimum practices for high-impact AI systems.
- NIST, Artificial Intelligence Risk Management Framework (AI RMF 1.0) (January 26, 2023) — the Govern, Map, Measure, Manage framework that structures organizational AI documentation.
- Office of Management and Budget, Uniform Administrative Requirements, Cost Principles, and Audit Requirements for Federal Awards (2 CFR Part 200) — single audit requirements; §200.501 threshold raised to $1,000,000 for fiscal years beginning on or after October 1, 2024 (89 FR 30046, April 22, 2024).
- CMS, MARS-E 2.2: Minimum Acceptable Risk Standards for Exchanges — security and privacy documentation standards for state-based Health Insurance Exchange systems established under the ACA.
