Write down the decision you are trying to make
“Can AI help us?” is too broad to evaluate. “Can a coordinator use an AI-prepared briefing to complete a service handoff with less correction work?” is a question with a recognisable output and an accountable reviewer.
The brief should identify the input information, expected output, permitted actions, exclusions and business owner. Explain what would count as useful progress and what would stop the exercise. A pilot cannot settle every future use case or integration.
- Agree the scope
- Build in sandbox
- Review and test
- Decide on production
Choose evidence before tuning the workflow
Use a representative set of scenarios: straightforward, incomplete, contradictory and outside scope. Keep some examples separate from the ones used to refine the instructions. Otherwise the team may become very good at demonstrating a familiar set without learning how the configuration handles new work.
Anthropic’s evaluation guidance emphasises that assessing agents involves their actions and resulting state, not only their final prose. Apply that principle to your acceptance review: did the intended record change happen, and did anything else change? [1]
An acceptance sheet your sponsor can understand
| Question | Useful evidence |
|---|---|
| Is the output supported? | Reviewers can trace important statements to the supplied information. |
| Can staff use it? | People can inspect, correct or reject the output without extra confusion. |
| Does the action behave correctly? | The permitted update succeeds; failed or disallowed work is reported honestly. |
| Is the effort worthwhile? | Total review, correction and processing work is compared with the baseline. |
| Can the team operate it? | An owner can monitor, pause and escalate problems. |
Set the actual thresholds with the people responsible for the process. There is no universal accuracy percentage that makes every workflow acceptable.
A worked example: missing-information follow-up
Illustrative scenario. A nonprofit team wants help preparing follow-up messages for incomplete applications. The pilot drafts a request for missing information using the organisation’s checklist. A reviewer checks the draft; the exercise excludes eligibility decisions and automatic outbound messages.
Include a complete application, a genuinely missing attachment, a document with conflicting details and a case the reviewer cannot access. A useful result explains why follow-up is needed without inventing a requirement or revealing unrelated information.
What REDA’s pilot includes—and where it ends
REDA’s assisted first sandbox pilot covers one agreed problem within a documented scope. Your administrator helps explain the data and processes and arranges the agreed sandbox access. The team reviews the pilot and carries out user acceptance testing before deciding whether to proceed.
Timing follows the scope and prerequisites. Production implementation, wider integrations and ongoing operating arrangements are scoped separately. Discuss a pilot with a clear question you want it to answer.
Sources
- Anthropic — Demystifying evals for AI agents · 9 January 2026. Accessed 22 September 2026.