Summary
A useful AI scribe pilot runs four to six weeks, with six to ten clinicians across at least two specialities, and it is decided by thresholds written down before it starts. Capture the baseline first, because without it the pilot can only be argued about. The criteria worth holding are note quality, the share of drafts accepted unedited, turnaround, time after clinic, and a counted hallucination rate.
Most scribe pilots fail on their design rather than on the product. They start without a baseline, run for two weeks, recruit the three clinicians most likely to enjoy it, and finish with a room full of impressions that nobody can hold a vendor to.
The published evaluations are the argument for doing it properly. A Cleveland Clinic study of ten primary care physicians over four weeks, published in Digital Health in 2025, found only one statistically significant change, a fall of 15,398 characters typed. Time outside scheduled hours went up by 13.7 minutes a day rather than down, and the authors named the absence of EHR integration as the reason the efficiency never arrived.
A larger trial reached the opposite conclusion on quality and a sobering one on safety. Across 100 clinicians and 7,499 consultations at Gold Coast Hospital and Health Service, published in BMC Health Services Research in January 2026, AI notes scored 37.06 of 40 on the PDQI-9 against 34.6 for clinician notes, and 58 percent of outputs were accepted into the record without modification. In the same trial 47 percent of staff reported hallucinations and 20 percent reported them frequently.
See it in your clinicWe will run the pilot against your own criteria, and hand back the numbers whether or not they favour us.
Request a demo| What you measure | How to measure it | A defensible threshold |
|---|---|---|
| Note quality | PDQI-9 scored blind by two clinicians on 20 notes | At or above your own current mean |
| Accepted unedited | Share of drafts entering the record with no change | Above 50 percent by week four |
| Turnaround | Consultation end to draft available | Under 5 minutes for 95 percent |
| Time after clinic | Minutes of documentation outside session hours | Falls against the baseline |
| Hallucination rate | Counted per 100 notes, any invented fact | Zero tolerated in medication or results |
| Coverage | Share of consultations the scribe captured at all | Above 95 percent of attempts |
| Clinician intent | Would you keep using this, asked in week one and week six | Rises rather than falls |
Capture the baseline first
The baseline is the only part of a pilot that cannot be recovered later. Two weeks before anything is installed, record the minutes each clinician spends documenting outside session hours, the time from consultation to a signed note, and the share of letters that leave within a week.
Score your own notes while you are at it. Twenty existing notes, scored blind on the PDQI-9 by two clinicians, gives the pilot a number to beat rather than a feeling to argue with. The Gold Coast trial used exactly this instrument, which is what allowed it to state that AI notes scored 37.06 against 34.6.
Without that fortnight, every claim afterwards is a comparison against memory, and memory in a busy clinic is generous about how long the admin used to take.
Who should be in it
Six to ten clinicians across at least two specialities, and they should not all be volunteers. A pilot staffed entirely by the enthusiastic tells you what the ceiling looks like and nothing about the floor.
Include at least one clinician who does not want to be there. Include the specialities with the hardest audio, which in most private practices means the ones with examination couches, ultrasound machines or three people in the room. If a product survives those, the rest of the clinic is straightforward.
Name one person who owns the pilot. Someone has to chase the weekly numbers, and a pilot without an owner produces an opinion rather than a decision.
Four to six weeks
Two weeks measures novelty. Clinicians are still learning what to say aloud, the templates are still wrong, and the numbers from week one describe a product nobody has learned yet.
Four to six weeks lets the learning curve finish inside the pilot rather than after the contract is signed. In the Cleveland Clinic study the median participant ran 61 encounters across four weeks, with a range from 1 to 181, which is a reminder that a headline average can hide a participant who barely used it.
Track usage per clinician weekly and treat a collapse in use as a result rather than as a data problem. A product people quietly stop opening has told you something.
Count the hallucinations
Nearly half the staff in the Gold Coast trial reported hallucinations and a fifth reported them frequently, in a trial that otherwise concluded the technology performed soundly. Both of those are true at once, and a pilot that only asks whether clinicians liked it will miss the second one.
Count them formally. Every clinician logs any invented fact, with the note and the date, and the log is read weekly rather than at the end. Separate the harmless from the dangerous, because a wrong pronoun and an invented dose are not the same finding.
Set the threshold before you see the data. Our own is that nothing invented in medication, dose, allergy or a result value is acceptable at any rate, and the clinician remains the author of every word that enters the record. Read our review of the hallucination evidence for what the wider research reports.
The failure the studies keep naming
The Cleveland Clinic authors attributed the missing efficiency to the absence of EHR integration. The draft was good and it still had to be moved by a person, which is where the saving went.
So a pilot has to test the whole path rather than the transcript. Where does the note land, who moves it, what happens to the letter that follows it, and how many clicks sit between the end of the consultation and a signed record. A scribe that produces an excellent draft into a window someone must copy from has not removed the work, it has relocated it.
This is the question to press hardest with any vendor whose product sits beside your record system rather than inside it.
What a pilot cannot answer
A pilot tells you about quality, speed and fit. It tells you nothing about what happens in year two, and the commercial questions have to be answered on paper before the pilot starts, not after it succeeds.
Get the renewal rate, the minimum term, what the price becomes when the trial cohort becomes the whole clinic, and what happens to your data if you leave. Our guide to the questions to ask a scribe vendor sets out the data protection half of that conversation.
Ask for the pilot's exit terms in writing too. A pilot you cannot walk away from without a fee is a purchase with a trial period attached, which is a different thing.
Expanding, and stopping
Expand when the written thresholds are met and the clinicians who did not want to be there would keep it. Those two together are a stronger signal than any single number, because a product that wins over its sceptics has cleared the real bar.
Stop when the thresholds are missed twice in a row after the learning curve should have finished, or when the hallucination log contains anything clinical that a person did not catch. Neither is a reason to abandon the category, and both are reasons to change the product.
Whatever the result, write down what the pilot measured and what it found. The next vendor conversation is much shorter when you can hand over a document instead of starting again.
Bring the criteria above to a demo and hold us to them on your own consultations.
Request a demoFrequently asked questions
How long should an AI scribe pilot run?
Four to six weeks. Two weeks measures novelty rather than the product, because clinicians are still learning what to say aloud and the templates are still wrong. In the Cleveland Clinic evaluation published in Digital Health in 2025 the median participant ran 61 encounters over four weeks, with a range from 1 to 181.
How many clinicians should take part?
Six to ten across at least two specialities, and not all of them volunteers. A pilot staffed by the enthusiastic shows you the ceiling and nothing about the floor. Include the specialities with the hardest audio, because a product that survives those makes the rest of the clinic straightforward.
What acceptance criteria should we set?
Note quality scored blind against your own current notes, the share of drafts accepted with no edit, turnaround from consultation to draft, documentation time outside session hours against a baseline, and a counted hallucination rate. Write the thresholds down before the pilot starts, because thresholds chosen after the data arrives are not thresholds.
What is a good rate of drafts accepted without editing?
Above 50 percent by week four is defensible. The Gold Coast Hospital and Health Service trial across 100 clinicians and 7,499 consultations, published in BMC Health Services Research in January 2026, found 58 percent of scribe outputs were accepted into the outpatient note without modification.
Do AI scribes actually save time?
Not automatically. The Cleveland Clinic study found time outside scheduled hours rose by 13.7 minutes a day rather than falling, and the only statistically significant change was 15,398 fewer characters typed. The authors named the absence of EHR integration as the reason. Test the whole path from consultation to signed record, not the transcript.
How should a pilot handle hallucinations?
Count them rather than survey for them. In the Gold Coast trial 47 percent of staff reported hallucinations and 20 percent reported them frequently, in a trial that concluded the technology performed soundly overall. Log every invented fact with its note and date, read the log weekly, and separate the harmless from anything touching medication, dose, allergy or a result value.
What should be agreed before the pilot rather than after?
The renewal rate, the minimum term, what the price becomes when the whole clinic uses it, what happens to your data if you leave, and the pilot's own exit terms. A pilot you cannot walk away from without a fee is a purchase with a trial period attached.
Does wai run pilots on these terms?
Yes, and against your criteria rather than ours. We will agree the thresholds in writing first, capture the baseline with you, and hand back the numbers whether or not they favour us. Clinicians still decide what enters the record.
Sources
Pilot it against your own criteria
Fifteen minutes on your workflows. If it is a fit we reply within 24 hours.
Partner with us
See it on your own patients, then build it with us
Thirty minutes on your own patients, doctors and rules, in your market. We reply within one working day, and the clinics that partner with us early shape what we build next.
See what wai does between the visits