The Knowledge Cliff: Test Recovery Before Critical System Expertise Disappears
Back to Signal
State & LocalModernizationGovernmentInfrastructure

The Knowledge Cliff: Test Recovery Before Critical System Expertise Disappears

September 9, 2026Jess Loban

What Washington auditors actually found

The Washington State Auditor's office published a finding in May 2026 with a practical warning for other states: the apportionment system supporting $30 billion in public education funding in the 2023–2025 biennium to roughly 300 school districts and charter and tribal schools statewide was described by a 2024 third-party review, cited in the audit, as at high risk of catastrophic failure. The software, described in the 2026 audit as seventeen years old, runs on aging infrastructure and lacks fully developed backup and recovery plans, and — the detail that should concern every state CIO in the country — depends on a vendor with only one trained staff member supporting its feeder systems, while OSPI lacks an IT technician with specific core-system knowledge.

OSPI had received the 2024 external warning before auditors made the finding public. The audit also reported test recalculations for three school districts in the 2023–2024 school year that found relatively minor discrepancies. That operating evidence matters: the concern is continuity and recovery exposure, not a finding that the system was universally calculating funding incorrectly.

How dependence accumulates

The Washington finding illustrates a transferable dependency risk. A state may operate systems that were procured a decade or two ago, handed to a vendor for implementation and support, and then left without a maintained knowledge-transfer plan. Potential contributors include staff turnover, missing knowledge transfer, absent or untested escrow rights, and a shrinking vendor support pool. Those are risks to investigate, not findings established for every state by this audit.

An agency can eventually become dependent on a support arrangement it cannot replace promptly, with recovery tasks that its own staff have not demonstrated. The outcome doesn't announce itself — it accumulates.

Ask questions uptime cannot answer

Routine service metrics can miss this dependency. A legacy system that processes transactions on time, passes routine reconciliation checks, and generates no helpdesk surge looks fine from the outside. A stronger annual IT assessment asks how many people — across both agency staff and vendor employees — could reconstruct the system from scratch if the current support team departed simultaneously. It also asks whether the vendor has formally documented recovery procedures that agency staff could actually execute.

It should examine the vendor's own operational continuity: whether that one trained employee has a designated backup, and whether the vendor organization itself is financially stable enough to survive the remaining years of the support contract. Washington OSPI received an external review in 2024, absorbed the warning, and obtained replacement funding beginning with the 2025–2027 budget; the interim control work still mattered while the replacement was being planned. The Legislature allocated sixteen million dollars across four fiscal years; a replacement system is targeted for the 2028-29 school year.

The system must continue supporting school funding during the replacement period; future allocations should not be assumed to equal the 2023–2025 amount.

Write continuity into the contract

The procurement decisions states make today influence the continuity risks their successors inherit. For critical systems, state buyers should consider vendor staffing minimums and succession documentation for critical-path support roles, source code escrow with usable retrieval and build rights where suitable, or equivalent data-export and transition protections for hosted services, and ongoing operational independence tests — periodic exercises in which agency staff demonstrate they can execute recovery procedures without vendor assistance. These provisions need to be matched to the service model and priced during acquisition.

An escrow deposit is of limited value if the state cannot build the code, obtain dependencies or lawfully run it; a hosted service may need a tested transition mechanism instead. Buyers should make any concession on continuity protections explicit. The difference between a legacy system and a ticking clock can be an untested dependency that never appeared in the acceptance criteria.

Protect the system during replacement

States that are now planning system replacements face a clear decision point: build operational independence requirements into replacement contracts, or repeat the cycle. OSPI's response in the audit targets a replacement for the 2028–2029 school year. OSPI's SASQUATCH procurement, issued November 10, 2025, scheduled proposals for February 13, 2026. The RFP already requires continuous knowledge transfer, effectiveness exercises, recovery objectives and scenarios, and a transition handoff. Those are meaningful contractual requirements to verify through delivery evidence. Its planned milestones are not proof that an award or deployment occurred. Before award, buyers have an opportunity to make continuity and transition terms part of competitive evaluation. What goes into a replacement contract can reduce the risk of recreating the same dependency.

The same holds true in every other state that is replacing a legacy system this year with another vendor-hosted, vendor-documented, vendor-sustained implementation. Neither the timing of the next audit nor the departure of the next key employee is predictable. Test continuity while the service is still running.

Test continuity before the next renewal

  1. Map the people dependency. Identify everyone who can change, restore and explain each critical component, including vendor staff. Record trained backups and the work that only one person can perform.
  2. Test recovery with a different operator. Give a qualified backup the runbook, approved credentials and test environment. Record missing steps, elapsed time and dependencies that required help.
  3. Review usable rights. Confirm access to data, configuration, documentation, build dependencies and transition assistance. Match escrow provisions to the actual product and licensing model.
  4. Fund interim controls. Do not wait for replacement go-live to document funding calculations, cross-train staff, test backups and review privileged accounts.
  5. Make independence an acceptance criterion. Require a successful handoff exercise and a maintained documentation set, with accountable owners on both sides of the contract.

What to measure

  • Key-person exposure: critical tasks with only one trained operator.
  • Recoverability: recovery exercises completed within the program's approved tolerance.
  • Transition readiness: documentation, export and build tests completed, with unresolved dependencies costed.

Test the handoff without the usual expert

Consider a hypothetical service with a documented recovery procedure that has only ever been executed by its original maintainer. A qualified backup operator receives the approved runbook and a controlled test environment. The test stalls because a build dependency, configuration value or external contact was never recorded. The document exists; the capability to use it has not yet transferred.

This exercise should test a defined task, not expect every agency employee to recreate the entire system. Select a recovery or maintenance action that is consequential to the service. Establish the required access and safety boundaries, then observe what the backup operator can complete without undocumented help. Record missing prerequisites and the time spent resolving them.

After correction, repeat the affected steps with the documentation the successor will actually receive. A demonstration by the person who wrote the instructions can conceal gaps because that person supplies missing context from memory. The evidence should show that an appropriately qualified replacement can perform the work.

Choose continuity arrangements that fit the service

Operational independence need not mean reproducing every supplier function inside the agency. For a hosted service, the practical protection may be a second supported recovery contact, defined transition assistance, usable exports and a tested route to another operating arrangement. For custom software, build rights and dependencies may be central. Match the protection to what the agency would need to do if the current arrangement ended.

A second supplier can reduce some dependencies while adding coordination and access-management work. Cross-training within the existing support team may be more practical for a bounded risk. A replacement project may address long-term exposure while leaving the current service dependent on the same people for several years. Compare those options without treating the existence of a new contract as evidence that continuity has been secured.

Have the program owner identify the maximum tolerable interruption and the essential work that must continue. Have engineering identify the dependencies and demonstrate the recovery steps. Have procurement confirm which rights, assistance and staffing commitments the agreement actually provides. Any gap between those three views belongs in an owned decision record.

The next renewal should carry evidence of usable knowledge transfer, not simply a statement that documentation was delivered. That gives leadership a practical basis for funding interim controls while longer-term modernization proceeds.

Sources and further reading

This is where Spartan X's engineering and program-execution disciplines belong together: making recovery, documentation and transition part of the deliverable, so a working system does not depend on one person's continued availability.

Share this article
LinkedIn

BUILD WITH US

Ready to Solve Hard Problems?

Spartan X builds AI systems, autonomous platforms, and cybersecurity solutions for defense and national security.