What the evidence says about failure
The pandemic exposed weaknesses in unemployment-insurance systems while agencies were implementing new programs and handling unprecedented demand. GAO's 2023 review identified staffing, contracting, management, financial and technical challenges across eight selected states. The sample was not nationally representative; its findings should not be turned into a claim that every modernization fails for the same reason.
The reviewed states also reported improvements, including greater stability and reduced paper processes. That combination matters: modernization can deliver value while still leaving difficult operational risks. The useful question is which risks the requirements and acceptance process make visible early enough to address.
Recover the rules before encoding them
UI decisions combine state law, federal requirements and administrative processes. Some implementation knowledge may reside in adjudicators' experience or in legacy code rather than current documentation. A platform selection made before that knowledge is examined can lock in assumptions the program has not agreed to defend.
Bring policy specialists, experienced staff and engineers together around representative cases. For each rule, identify its authority, effective period, input data, exceptions and expected outcome. Preserve legitimate judgment rather than forcing every ambiguous case into a binary automated answer.
The result should be a versioned set of examples and tests that can be revisited when policy changes. A rule catalogue that sits apart from the implementation will quickly lose value. The program needs someone accountable for maintaining both the interpretation and the evidence that the system follows it.
Test the surge across the whole journey
An application server that handles increased traffic can still feed a bottleneck in identity verification, document review, payments or the call center. Load testing should follow a claimant through dependencies and exceptions, not stop at successful login.
- Demand: concurrent users, document submissions, status checks and repeated attempts after a failure.
- Dependencies: slow or unavailable verification, payment and notification services.
- People: queues requiring adjudication, support or fraud investigation when volumes rise.
- Recovery: restarting partial work without duplicate payments, lost documents or contradictory notices.
Use scenarios grounded in the state's history and planning assumptions. Buyers need a justified load model, meaningful acceptance criteria and a contract that assigns responsibility for correcting failed tests.
Use AI where its work can be checked
Document extraction, routing and anomaly detection can assist staff, but their suitability has to be established for the specific task. A fluent explanation does not prove a model read the right record or applied the current rule. Compare the proposed automation with a defensible baseline and examine the cases where reviewers disagree.
The risk increases when a recommendation becomes a denial, a fraud finding or a payment action. Define the evidence, review, notice and appeal requirements for that action before delegating any part of it. Keep logs of source information, system versions and material human decisions so an affected case can be reconstructed.
Historical decisions are not an unquestionable gold standard. They can contain mistakes or unequal treatment. Evaluation should therefore include policy review and appropriately independent judgment, rather than simply rewarding a model for matching past outcomes. Accuracy, timeliness and resident access all belong in the assessment.
Make a rule change part of the acceptance exercise
In a hypothetical acceptance test, a policy change takes effect during a claimant's ongoing case. The team must show which rule applies to each relevant period and how the system handles an adjustment to information submitted earlier. The test is not satisfied merely because the new rule produces the expected answer on a newly created case.
Program specialists establish the authoritative interpretation and expected results. Engineers trace the affected inputs, calculations and notices. Operations staff test the exception path when information is incomplete or disputed. Preserve effective dates and versions so that a later reviewer can reconstruct why the system reached a particular result.
Include a correction after an initial decision. Can staff amend the right record without silently changing unrelated periods? Do dependent calculations and notices remain consistent? Where a human decision is required, is the case routed to someone with the information and authority to resolve it? These are recommended acceptance questions; the applicable law and program requirements determine the actual decision and process.
This exercise can expose whether a proposed configurable rules engine truly separates policy from application code. It can also show that a simpler change process is adequate for a particular system. Buy the demonstrated ability to implement and verify a justified change, not a general claim that the product is flexible.
Test the backlog after the traffic subsides
A surge is not over when the website becomes responsive again. In an illustrative scenario, document submissions and verification requests accumulate while a dependency is unavailable. Once it returns, the system must reconcile partial work, avoid duplicate actions and route unresolved cases without losing their history.
Measure the recovery of the queues as well as peak traffic. A platform can accept many applications while creating more exception work than staff can clear. Identify which cases are waiting, what information is missing and what service commitment applies. Do not treat a completed upload as a completed claim determination.
The remedy depends on the bottleneck. Additional computing capacity may help a saturated component; it cannot supply an adjudicator's judgment. Better retry handling may prevent duplicate work; it cannot make an unavailable source authoritative. A narrower automation role, improved notices or temporary support capacity may be more useful than another platform feature.
Before acceptance, the program and technical owners should agree on the load assumptions, evidence and unresolved limits. Record what would cause the agency to delay cutover, reduce scope or use a continuity procedure. A successful test supports a decision within its tested conditions; it does not guarantee performance under every future emergency. Maintaining the test cases and updating the assumptions gives the agency something more durable than a one-time demonstration.
Give acceptance criteria an operational owner
GAO recommended stronger pilot design and performance measurement. Its recommendation record notes that DOL published customer-experience standards and metrics in December 2024 and encouraged states to use them; the record does not describe that encouragement as a universal mandate. The broader performance-measure recommendation remained open in GAO's April 2026 update. That is useful progress, while the continued need to measure real operations remains relevant to state buyers. State requirements should reflect that newer guidance alongside the lessons from earlier failures.
- Document the policy baseline. Create traceable rules and representative cases, with named owners for unresolved interpretations.
- Define the stressful cases. Include demand surges, policy changes, unavailable services, accessibility barriers and correction of erroneous records.
- Test before acceptance. Require observable outcomes, not only a demonstration of features. Record failures and retest the changed behavior.
- Preserve accountable decisions. Separate automated assistance from consequential determinations and verify the review and recourse path.
- Maintain the service after launch. Fund monitoring, rule updates, incident response, training and knowledge transfer.
The modernization should leave the agency better able to respond to the next change. That capability comes from a maintained connection between policy, software and operations—not from a one-time successful cutover.
Sources and further reading
- GAO: DOL Needs to Further Help States Overcome IT Modernization Challenges — July 2023 findings, sample limitations and subsequent recommendation updates. The delivery sequence above is our analysis.
Spartan X's engineering, AI consulting and program-execution disciplines meet in this work: translating program rules into testable behavior and making the operating requirements part of the delivery plan from the start.
