What the evidence says about failure
The pandemic exposed weaknesses in unemployment-insurance systems while agencies were implementing new programs and handling unprecedented demand. GAO's 2023 review identified staffing, contracting, management, financial and technical challenges across eight selected states. The sample was not nationally representative; its findings should not be turned into a claim that every modernization fails for the same reason.
The reviewed states also reported improvements, including greater stability and reduced paper processes. That combination matters: modernization can deliver value while still leaving difficult operational risks. The useful question is which risks the requirements and acceptance process make visible early enough to address.
Recover the rules before encoding them
UI decisions combine state law, federal requirements and administrative processes. Some implementation knowledge may reside in adjudicators' experience or in legacy code rather than current documentation. A platform selection made before that knowledge is examined can lock in assumptions the program has not agreed to defend.
Bring policy specialists, experienced staff and engineers together around representative cases. For each rule, identify its authority, effective period, input data, exceptions and expected outcome. Preserve legitimate judgment rather than forcing every ambiguous case into a binary automated answer.
The result should be a versioned set of examples and tests that can be revisited when policy changes. A rule catalogue that sits apart from the implementation will quickly lose value. The program needs someone accountable for maintaining both the interpretation and the evidence that the system follows it.
Test the surge across the whole journey
An application server that handles increased traffic can still feed a bottleneck in identity verification, document review, payments or the call center. Load testing should follow a claimant through dependencies and exceptions, not stop at successful login.
- Demand: concurrent users, document submissions, status checks and repeated attempts after a failure.
- Dependencies: slow or unavailable verification, payment and notification services.
- People: queues requiring adjudication, support or fraud investigation when volumes rise.
- Recovery: restarting partial work without duplicate payments, lost documents or contradictory notices.
Use scenarios grounded in the state's history and planning assumptions. Buyers need a justified load model, meaningful acceptance criteria and a contract that assigns responsibility for correcting failed tests.
Use AI where its work can be checked
Document extraction, routing and anomaly detection can assist staff, but their suitability has to be established for the specific task. A fluent explanation does not prove a model read the right record or applied the current rule. Compare the proposed automation with a defensible baseline and examine the cases where reviewers disagree.
The risk increases when a recommendation becomes a denial, a fraud finding or a payment action. Define the evidence, review, notice and appeal requirements for that action before delegating any part of it. Keep logs of source information, system versions and material human decisions so an affected case can be reconstructed.
Historical decisions are not an unquestionable gold standard. They can contain mistakes or unequal treatment. Evaluation should therefore include policy review and appropriately independent judgment, rather than simply rewarding a model for matching past outcomes. Accuracy, timeliness and resident access all belong in the assessment.
Give acceptance criteria an operational owner
GAO recommended stronger pilot design and performance measurement. Its current recommendation record notes that DOL published customer-experience standards and metrics in December 2024. That is useful progress, while the continued need to measure real operations remains relevant to state buyers. State requirements should reflect that newer guidance alongside the lessons from earlier failures.
- Document the policy baseline. Create traceable rules and representative cases, with named owners for unresolved interpretations.
- Define the stressful cases. Include demand surges, policy changes, unavailable services, accessibility barriers and correction of erroneous records.
- Test before acceptance. Require observable outcomes, not only a demonstration of features. Record failures and retest the changed behavior.
- Preserve accountable decisions. Separate automated assistance from consequential determinations and verify the review and recourse path.
- Maintain the service after launch. Fund monitoring, rule updates, incident response, training and knowledge transfer.
The modernization should leave the agency better able to respond to the next change. That capability comes from a maintained connection between policy, software and operations—not from a one-time successful cutover.
Sources and further reading
- GAO: DOL Needs to Further Help States Overcome IT Modernization Challenges — July 2023 findings, sample limitations and subsequent recommendation updates. The delivery sequence above is our analysis.
Spartan X's engineering, AI consulting and program-execution disciplines meet in this work: translating program rules into testable behavior and making the operating requirements part of the delivery plan from the start.
