Anonymised composite case study
This case study reflects recurring patterns across supplier-led testing programmes. It does not identify a single client, system or delivery team.
A transformation programme reported high test execution and pass rates as it approached user acceptance testing. Dashboards showed thousands of completed tests, falling defect numbers and improving daily throughput. Governance interpreted the trend as evidence that the solution was nearing acceptance.
The problem
The programme measured activity more reliably than risk coverage. A passed test demonstrated that one scripted check had produced an expected result in a particular environment. It did not prove that the critical business processes, interfaces, migrated data and operational controls were ready.
User acceptance testing was also being positioned as the final opportunity to find anything missed earlier. That transferred supplier system-testing obligations to business users and placed acceptance under the greatest schedule pressure.
What the evidence showed
- Requirements coverage was reported, but traceability to priority business and operational risks was incomplete.
- Pass rates mixed critical end-to-end tests with low-consequence checks.
- Blocked and not-run tests were excluded from headline percentages.
- Environment limitations meant some interfaces and batch processes had not been exercised realistically.
- Defect closure relied on workflow status without consistent retest evidence.
- Severity decisions considered individual defects but not cumulative process impact.
- UAT entry criteria were being relaxed because the planned start date had become fixed.
The available evidence did not justify the confidence implied by the dashboard.
The assurance intervention
Enigma reset the test evidence around the acceptance decision. The purpose was not to increase test volume. It was to establish whether the right risks had been tested under credible conditions and whether the results could withstand challenge.
The intervention introduced:
- A risk-to-requirement-to-test evidence chain for critical business outcomes.
- Separate reporting for passed, failed, blocked, not run and not testable scope.
- Explicit environment and data limitations attached to every affected conclusion.
- Closure verification requiring retest evidence and regression impact assessment.
- End-to-end scenarios covering interfaces, migrated data, roles, controls and operational exceptions.
- Hard UAT entry criteria distinguishing supplier test completion from business acceptance activity.
What changed
Governance stopped asking only how many tests had passed. It asked which material risks were proven, which remained exposed and what limitations prevented a reliable conclusion.
Several apparently healthy metrics were reclassified. Blocked critical tests became visible rather than disappearing from the denominator. Defects could not be treated as closed without evidence. UAT was protected from becoming late supplier testing, and business users received a stable, defined basis on which to assess fitness for use.
The supplier retained ownership of test delivery. Independent assurance verified the sufficiency and integrity of the evidence without taking over execution.
The outcome
The board received a more cautious but more useful position. Confidence was tied to risk coverage and evidence quality rather than volume. Decisions could distinguish between a recoverable evidence gap, a controlled residual risk and a failure that prevented progression.
The central lesson is that test activity is not a proxy for acceptance. Metrics become meaningful only when their scope, exclusions, environment, data, defects and business consequence are explicit.
Enigma provides quality and test assurance for organisations that need an independent view of whether testing supports the decision being made.