What decision is the pilot meant to inform?
Write the decision before choosing participants. The team may need to know whether a shared prompt helps administrators produce a clearer handover, or whether an approved document pack supports more consistent brief preparation. A pilot designed only to show that AI can generate text will not answer whether the workflow belongs in daily work.
Choose a task that recurs often enough to observe but is small enough to inspect. Identify its input, expected output and responsible approver. Exclude decisions that participants cannot safely reverse or judge. Begin with invented material if the organisation has not approved real information for the proposed account and purpose.
The Privacy Commissioner’s guidance is relevant when AI use involves personal information1. Resolve the information boundary with the appropriate people before collection starts. A pilot label does not remove those questions. Record which examples are permitted and what participants should do if they encounter information outside the agreed scope.
How do we define a usable baseline?
Ask someone familiar with the task to complete a representative example using the current method. Record the work involved, including clarification, drafting, checking and correction. Keep the input and the accepted output so the later comparison concerns similar work. A vague estimate of how long the task usually takes is useful context but weaker evidence.
Describe the conditions that affect difficulty. A complete brief is easier than one with missing dates or conflicting instructions. If the AI-assisted attempt uses cleaner input, record that change. The team may still benefit from improving the source process, but it should not attribute every improvement to the tool.
Decide how timing will be recorded without turning observation into a burden. Separate active work from waiting for another person. The purpose is to understand effort, not to rank colleagues. Explain that participants should report failed drafts and difficult checks honestly, because those observations help determine whether the workflow is suitable.
What should the pilot brief include?
Name the task owner, participants, approved input, account environment, review standard and decision point. Describe what is outside the pilot. For example, an exercise might permit drafting an internal handover while excluding external sending, account changes and live customer records. These limits should be readable before anyone begins.
Write a concrete acceptance rule. An action list might need every owner and deadline to be supported by a source note, with missing details clearly flagged. A summary might need each material claim linked to a passage. Avoid a rule such as looks professional, which judges presentation while leaving accuracy and completeness unclear.
Include a pause condition and an escalation contact. If the task unexpectedly requires restricted information or the output cannot be checked, the participant should know what to do. The pilot owner can revise the scope or choose another example. Continuing despite an unresolved boundary produces poor evidence and may create an avoidable problem.
How do we run an individual attempt?
Keep the input unchanged and record the prompt used. Note the account experience and any feature limitation that affects the attempt. Ask for a bounded draft with visible assumptions. Do not keep improving the source or coaching the model in ways that disappear from the record; those steps consume effort and affect the result.
Review the draft using the agreed standard. Mark unsupported statements, omissions and changes in meaning. Then record the correction work needed to reach an accepted output, or mark the attempt unusable. A result can be helpful in one part of the task and unhelpful in another, so retain a short explanation alongside the timing fields.
Ask another participant to repeat the method on a different invented example. This tests whether the prompt and checking instructions can be handed over. If the method works only when its creator supplies extensive unwritten context, the team has more preparation to do before treating it as a reusable workflow.
Which failures should we deliberately test?
Include an incomplete source, a contradiction and a request outside the task’s authority. For an office handover, leave a deadline blank and change an earlier instruction in a later note. The correct checking behaviour is to preserve the gap and identify the later evidence, not to produce a perfectly complete-looking schedule.
For a data summary, include a reversal or a duplicated-looking reference and inspect what happens. Participants should explain the issue with row references before proposing a correction. Do not assume that a successful ordinary example establishes reliable behaviour when the data is awkward. The difficult cases often reveal whether the review method is workable.
Record how a failure was caught. If only a specialist noticed a subtle error, the eventual workflow may require that specialist’s review. If a simple checklist reliably exposes the issue in the exercise, retain the checklist and test it again. The pilot should teach the organisation what oversight actually costs.
How do we compare the evidence fairly?
Look at accepted outputs, total effort and the kind of errors found. Keep drafting time separate from checking and rework. A quick first answer can still produce a slower completed task. Conversely, a modest time difference may be useful if the output is easier to review or the process makes missing information visible earlier.
Do not turn observed minutes into a cash saving without examining what changes in the business. Released capacity is not automatically a reduced expense. Ask the manager how any useful time would be used and whether the measured task occurs often enough to matter. Keep that discussion separate from the raw observations.
Inspect the examples behind the average. A small set can contain unusually easy or difficult tasks, and a single serious error may matter more than a favourable mean. Describe the limitations plainly. This guide supplies a measurement method, not a result or a promise that an AI pilot will produce a return.
| Observation | Record | Interpretation limit |
|---|---|---|
| Accepted result | Reviewer decision and checked output | A fluent draft may still fail |
| Effort | Baseline plus assisted drafting and review | Different inputs can distort the comparison |
| Failure | Error and the check that caught it | Rare serious errors still matter |
| Capacity | What useful work can follow | Time released is not automatically cash saved |
What decisions can the review meeting make?
The options include continuing the bounded trial, revising the prompt or source process, changing the review arrangement, expanding to a closely related task, or stopping. State the reason and the evidence supporting it. A decision to stop can be a useful outcome when it prevents a larger investment in an unsuitable workflow.
Before expansion, ask whether the new users, information or output destination change the approval conditions. A prompt tested on fictional notes does not establish permission to process live employee records. A draft used internally may need different review before it becomes external correspondence. Revisit those changes with the relevant owners.
Assign responsibility for maintaining the prompt, sources and review sheet. If nobody owns them after the demonstration, the method can drift as documents and products change. Set a review trigger tied to the workflow and retain a simple way for users to report a recurring problem or an unexpected output.
Where does training fit into the pilot?
Training can prepare participants to write bounded requests, inspect sources and record the effort needed to correct an answer. It can also help the group agree what a useful review looks like. Those skills should support the pilot’s decision, rather than substitute for evidence collected on the organisation’s own permitted tasks.
Bring an invented example and the intended acceptance rule to a workshop. Ask the trainer to help participants practise both ordinary and failure cases. Keep a checked example, a flawed example and the review notes so colleagues can understand the method afterwards. A collection of prompts alone does not explain how the outputs should be assessed.
The ChatGPT rollout and responsible-AI courses below are relevant starting points. Describe the proposed task, account setup and decision owner in your enquiry. We can discuss a session that helps the team prepare and evaluate a bounded trial while leaving organisational approvals and purchasing decisions with the people authorised to make them.