Keep the contract dispute separate from the engineering question
Calls to minimize refusals require buyers to distinguish requests the deployed system should complete, decline, or return for clarification. An external report on Pentagon–OpenAI contracting is a research lead for this discussion. We could not independently inspect that report or the underlying agreement, so it is not evidence here of an executed contract clause. An integrator should obtain the applicable executed terms before drawing conclusions about its obligations or safeguards.
OpenAI's own public account describes a layered approach: cloud deployment, a retained safety stack, cleared personnel, and contractual restrictions. That is the company's account of its arrangement, not an independent finding that every protection will work as intended. It is nevertheless direct evidence against assuming that OpenAI publicly agreed to provide an unrestricted model. OpenAI's agreement statement
The disagreement with Anthropic illustrates why vendor policy belongs in the acquisition discussion. Anthropic identified mass domestic surveillance and fully autonomous weapons as two uses it would not support under the requested terms. OpenAI also publicly described restrictions in these areas, while explaining a different approach to enforcement. These statements do not support a simple conclusion that access goes only to vendors that abandon safeguards. They do show why buyers must examine actual terms and deployment controls rather than infer them from a provider's participation. Anthropic's statement
The department's May 1 announcement names eight classified-network partners: SpaceX, OpenAI, Google, NVIDIA, Reflection, Microsoft, Amazon Web Services, and Oracle. It identifies planned IL6 and IL7 deployments and reports more than 1.3 million GenAI.mil users at that point. Neither that provider list nor the platform's scale establishes identical contract terms, safeguards, or availability at every classification level. Department classified-network announcement
Four properties that should not share one score
A model can answer every question and still perform badly. It can also refuse a legitimate request because its filters mistake context for prohibited conduct. Both problems matter, but they require different tests.
- Task accuracy: Is the response correct, supported by the available information, and suitable for its intended use?
- Appropriate refusal: Does the system decline requests it must not fulfill while permitting legitimate work?
- Authorization: Is this user allowed to access the data or initiate the requested action?
- Operational control: Can responsible personnel understand, constrain, monitor, and stop the system's actions?
A low refusal rate answers only part of the second question. It does not establish the other three. Nor can a model's willingness to answer authorize an action that the user or application is not permitted to take.
The distinction matters especially in decision support. A fluent answer can become influential even when a human formally remains responsible. An effective review process needs relevant evidence, enough time, and authority to challenge the output. A nominal approval step alone provides little information about the quality of that review.
What ARMOR measures
The ARMOR 2025 authors report evaluating 21 models using 519 multiple-choice questions grounded in military doctrine; the displayed results table lists 20 model entries, a reporting discrepancy to keep in view. It reports accuracy and False Refusal Rate (FRR), defined around refusals on the benchmark's doctrinally valid prompts. The coding also treats answers outside the required A/B/C format as refusals, so the metric does not isolate safety-filter behavior. Several evaluated models recorded no refusals under that method, while accuracy varied. ARMOR research paper
That is a useful, bounded result. A zero-refusal score on a curated question set does not mean a model will accept every real request, identify every unlawful request, or operate safely in a deployed workflow. The benchmark tests responses to questions; it does not certify a complete operational system. Its results also belong to the particular models, versions, and test conditions studied.
Procurement teams can use such research to improve evaluation design. They should resist turning one metric into a universal ranking. An accurate answer, an appropriate refusal, a justified request for more information, and an unsupported answer should receive different treatment in a test that reflects the proposed application.
Policy still requires assurance and accountability
The June 5 NSPM-11 directs national-security AI adoption alongside assurance, security, and accountable use. It calls for testing, evaluation, validation, and verification, and retains responsibility with commanders and agency leadership. It also directs an update to DoD Directive 3000.09; the direction itself is not evidence that every implementing document has been issued. NSPM-11
These responsibilities cannot be reduced to whether a model says yes or no. The allocation of legal duties and liability depends on the applicable law, contract, and deployment. An integrator should not assume that a rumored foundation-model clause automatically transfers liability or removes a control in its own architecture.
The engineering responsibility is more concrete: identify which component enforces each requirement and produce evidence that the combined system does so. Model behavior, identity and access controls, application permissions, human review, monitoring, and contractual restrictions may each contribute. A failure in one layer should not silently invalidate every other safeguard.
Write a specification that can be tested
- Name the use. Define users, data, permitted tasks, and consequences of error before choosing a behavioral threshold.
- Separate failure categories. Measure wrong answers, inappropriate refusals, unsafe compliance, missing evidence, and unauthorized actions independently.
- Test the deployed configuration. Evaluate the actual model version, retrieval sources, application controls, and user workflow together.
- Assign responsibility for changes. Require notice and reassessment when model updates or application changes could affect previous findings.
- Preserve the evidence. Retain evaluation results, accepted limitations, approvals, and a practical process for reporting and correcting failures.
Shared AI services can reduce duplication, but shared access does not make every deployment identical. A logistics application and a sensitive analytical workflow may use related technology while needing different permissions, evidence, and review. Their specifications should preserve those distinctions.
Useful AI should complete legitimate work reliably. That is a stronger purchasing objective than simply minimizing refusals, and it gives both buyers and suppliers a clearer basis for demonstrating performance.
Make the denominator part of the purchasing decision
A refusal score can change because the test population changed, even if the model did not. A set dominated by straightforward permitted questions answers a different evaluation question from one that includes incomplete instructions, requests outside the user's authority, or genuinely prohibited uses. Buyers need the composition of the evaluation alongside the reported rate.
For a hypothetical administrative assistant, a correct response might be a completed summary when the source record is available, a request for clarification when the requested period is ambiguous, or a refusal to disclose a record the user cannot access. Treating all three as an answer-versus-refusal contest would penalize some useful behavior and reward some unacceptable behavior. The acceptance criteria should recognize the difference before the supplier sees the test results.
This does not mean every cautious answer deserves credit. An assistant that repeatedly asks unnecessary questions can waste staff time and make a legitimate workflow unusable. The review should distinguish a justified clarification from avoidable friction, using the task's actual information requirements. That is a more precise improvement target than instructing a supplier simply to make the model less reluctant.
A defensible comparison keeps task conditions sufficiently comparable and explains remaining differences. If one product has access to a better reference collection or narrower permitted scope, its result should not be attributed solely to its foundation model. Buyers are acquiring the deployed system's usefulness, but they still need to understand which part of the system produced the improvement.
Leave room to improve without losing the acceptance basis
A specification should allow corrections while preserving the evidence behind approval. If an update reduces inappropriate refusals but increases unsupported answers, the improvement is not established by the first metric alone. The responsible team needs to examine the tradeoff in the intended use and decide whether the new configuration remains acceptable.
A limited deployment can be a reasonable alternative to accepting an unresolved behavior across an entire organization. So can narrowing the permitted task or improving the source material before changing the model. These choices have costs: reduced reach, additional review, or more preparation work. State those costs alongside the expected benefit rather than portraying every restriction as failure or every expansion as progress.
The next supplier discussion should center on a small set of consequential, representative cases and the evidence needed to judge them. Ask how a disputed result will be investigated and how the buyer will be notified when the relevant behavior changes. That conversation turns a politically charged metric into a concrete question of reliable service and accountable acquisition.
Sources and further reading
- OpenAI: public account of its agreement
- Anthropic: statement on department discussions
- Department: classified-network AI agreements
- ARMOR 2025: benchmark methods and findings
- White House: NSPM-11
Spartan X's AI consulting and cybersecurity practices connect model evaluation with the permissions, evidence, and human responsibilities surrounding an application. That work gives acquisition teams a defensible specification and gives operators a clearer understanding of the system they are being asked to trust.
