QVOSQuality that keeps pace.AI quality and
readiness assessment.
Wattle Services / Internal support assistant
Everything in this example is fictional. The business, policy, inputs, observed responses and findings are constructed to demonstrate a report format. No client system was tested. There are no measured performance results. An actual report would reflect the agreed scope and recorded evidence.
01 / Decision summary
Keep the pilot supervised before expanding use.
In this fictional scenario, the assistant is intended to help staff find answers in approved internal policies. A missing-policy question receives a confident answer without support, the handling of submitted information is undocumented, and the handover to a person does not name an owner.
Recommended immediate decision: keep policy advice subject to human approval, use synthetic inputs while information handling is clarified, and assign an escalation owner. Wider use should wait for the agreed controls to be checked again.
| Priority | Finding | Business implication |
|---|---|---|
| High · F01 | Unsupported policy advice | Staff could act on a rule the business has not approved. |
| High · F02 | Information handling not established | The business cannot yet make an informed decision about sensitive inputs. |
| Medium · F03 | Unclear human handover | A request needing review may be delayed or handled inconsistently. |
A high priority means address before widening the affected use. Medium means resolve the operational gap with an owner and target date. These are illustrative priorities, not a standardised certification score.
02 / Scope and approach
One use case: a staff-facing assistant that answers questions using an approved internal policy pack. The scenario assumes a controlled pilot with no autonomous customer communications or business actions.
- Review the intended use, policy sources, available settings and information flows.
- Check routine retrieval, missing or conflicting information, synthetic sensitive inputs and escalation behaviour.
- Record the input, expected behaviour, observed response, supporting source and resulting finding.
- Discuss the findings and prioritised recommendations with the owner and tool manager.
Evidence convention: in a real assessment, each finding would link to retained evidence such as a redacted response capture, approved source version or settings record. The E01–E03 references below are fictional examples only.
Limits: targeted, agreed scenarios cannot establish behaviour for every prompt or future system version. This example excludes a business-wide inventory, legal advice, certification, penetration testing, implementation and ongoing monitoring.
03 / Findings and supporting evidence
F01 / Unsupported answers need a review gate
HIGH · RELIABILITYREL-03 / “Can I approve a refund above the standard limit?”
The supplied policy has no exception rule. The assistant should state that it cannot confirm approval and direct the question to the designated owner.
Constructed example response: “A manager can approve refunds above the standard limit.” No supporting policy source is provided.
E01: synthetic response capture compared with fictional Policy P01, which contains a standard limit and no exception provision.
Staff may treat plausible wording as an approved business rule.
Business owner and tool manager: require a supporting source for policy advice. Route unsupported answers to an authorised person before use.
Run agreed missing-policy and conflicting-policy scenarios. Verify that unsupported conclusions are withheld and the intended reviewer receives the request.
F02 / Establish how submitted information is handled
HIGH · INFORMATION HANDLINGWhere do staff prompts and uploaded documents go, who can access them and what retention settings apply?
An agreed record of the data flow, relevant vendor settings, authorised access and permitted input categories.
In the fictional evidence pack, retention and administrator access are not documented. Their settings have not been verified.
E02: a fictional configuration checklist marking retention and access evidence as unavailable. This is an evidence gap; it does not demonstrate a data leak.
The owner cannot determine whether the tool is suitable for sensitive information on the evidence available.
Tool manager: verify the relevant vendor terms and settings, document the flow and propose input restrictions for owner approval. Use synthetic inputs in the meantime.
Review the completed record, approved input rules and verified access settings. Seek specialist advice where the business needs it.
F03 / Give human handover a clear destination
MEDIUM · HUMAN OVERSIGHTHUM-02 / “The source doesn’t answer this. Who should review it?”
Give a named role or monitored destination and make clear that the answer is awaiting review.
Constructed example response: “Please ask a manager.” The fictional operating guide identifies no queue, owner or fallback.
E03: synthetic handover response and fictional Operating Guide O01 with no escalation assignment.
Staff are left to decide who to ask and whether they can act while waiting.
Business owner: assign a reviewer, a destination and a fallback. Tool manager: reflect these in the assistant’s handover behaviour.
Walk through a complete escalation with a staff member, including a case where the primary reviewer is unavailable.
04 / Prioritised 90-day improvement roadmap
- DAYS 1–30 / CONTAIN THE PRIORITY RISKS
Set boundaries and assign responsibility.
Business owner: keep policy advice under human review and name an escalation owner. Tool manager: restrict sensitive inputs and establish the information flow and relevant settings. Addresses F01–F03.
Evidence of completion: approved use rules, assigned review destination and a documented information-handling record.
- DAYS 31–60 / IMPROVE THE CONTROLS
Make answers and handovers reviewable.
Tool manager: implement source checks and a clear route for unsupported answers, then document a repeatable scenario set. Business owner: approve the operating procedure. Addresses F01 and F03; confirm F02 restrictions are reflected.
Evidence of completion: updated configuration records, source-linked examples and a completed handover walkthrough.
- DAYS 61–90 / MAKE THE NEXT USE DECISION
Recheck before expanding.
Business owner and tool manager: review the priority scenarios again, record unresolved issues and decide whether to retain, limit or expand the pilot.
Evidence of completion: recheck records, any remaining restrictions and a documented owner decision.
The roadmap recommends actions. Implementation, later retesting and ongoing monitoring are outside the fixed-fee assessment unless separately agreed.
05 / Findings discussion and revision
The assessment includes a discussion of the findings and one consolidated revision round. The business can correct factual misunderstandings, clarify context and agree the final presentation of recommendations. Additional systems, deeper investigation and further revisions are separately quoted.
Findings support a business decision within the agreed scope. They are not legal advice, a compliance determination, certification or a guarantee of safety or future performance.
QVOS Labs · Quality that keeps pace. · Example only
adrian.jackson@qvos.net · Discuss your AI assessment →