Key points
- The PDQI-9 scores nine attributes of a note from 1 to 5, and showed inter-rater reliability of 0.83 in its validation study.1
- In a vendor-authored study of 12,999 clinician-annotated sentences, drafts had a 1.47% hallucination rate and a 3.45% omission rate, and 44% of hallucinated sentences were major.2
- When three language models scored notes with a modified PDQI-9, their agreement was poor, with a Krippendorff's alpha of minus 0.433, so clinicians do the scoring.4
- wai drafts the note during the consultation, and the clinician reviews, edits and signs it, which is where this scoring belongs.5
Summary
Score a sample of AI-drafted notes against the consultation with two tools. The PDQI-9 rates nine qualities of the note from 1 to 5, and a sentence-level error count records hallucinations by type and omissions, each marked major or minor. Use two clinician raters on a shared subset, at least 30 notes per note type, and report the counts with their denominators.
A month of structured scoring shows where an AI-drafted note needs your eye first, so every signature goes on a note you have measured. The method below takes an afternoon a week.
See it in your clinicThirty minutes on your own patients and doctors.
Request a demoRun a consultation with wai, then score the drafted note yourself with the rubric on this page before you sign it.
| Check | What the rater asks | How to score |
|---|---|---|
| Up-to-date | Does the note reflect the latest information from this consultation | 1 to 5 |
| Accurate | Is every statement true to the consultation | 1 to 5 |
| Thorough | Is everything clinically relevant included | 1 to 5 |
| Useful | Would the next clinician find what they need | 1 to 5 |
| Organised | Is information in the expected sections and order | 1 to 5 |
| Comprehensible | Is it clear to another clinician | 1 to 5 |
| Succinct | Is it without padding and repetition | 1 to 5 |
| Synthesised | Does it pull findings together into an assessment | 1 to 5 |
| Internally consistent | Do the sections agree with each other | 1 to 5 |
| Fabrication | Is anything stated that the consultation does not support | Count, major or minor |
| Negation | Is any relevant fact reversed | Count, major or minor |
| Causality | Does the draft speculate on a cause not stated | Count, major or minor |
| Contextual | Are unrelated topics mixed together | Count, major or minor |
| Omission | Is any clinically relevant statement missing | Count, major or minor |
The PDQI-9, for overall quality
The Physician Documentation Quality Instrument was refined to nine items, up-to-date, accurate, thorough, useful, organised, comprehensible, succinct, synthesised and internally consistent, each scored on a five-point scale.1 In its validation, 31 physician raters scored admission notes, progress notes and discharge summaries, with internal consistency of 0.87 to 0.94 and inter-rater reliability of 0.83.1
A Veterans Health Administration study used a modified PDQI-9 with 30 blinded raters to compare notes from 11 AI tools and 18 human note takers on five standardised primary care cases, scoring ten domains out of 50.3 Human notes scored higher in every case, with the largest gaps in thoroughness and organisation.3 Score your own notes before relying on any published figure.
Hallucinations and omissions, counted separately
The PDQI-9 rates descriptive qualities of the note as a whole and is not designed to check for specific note components.1 A sentence-level pass against the consultation catches individual errors. One published framework, from authors employed by a documentation vendor, labels each sentence of the draft against the transcript and records two error classes.2
A hallucination is content the consultation does not support. The framework splits hallucinations into four types, fabrication of information not in the consultation, negation of a clinically relevant fact, causality where the draft speculates on a cause, and contextual errors that mix unrelated topics.2 An omission is a clinically relevant statement from the consultation that is missing from the note.2
Each error is marked major if it could change diagnosis or management if left uncorrected, otherwise minor.2 Across 450 notes in that vendor-authored study, fabrications were 43% of hallucinations and negations 30%, and the plan section held the largest share of major hallucinations, so read the plan first.2 Published rates across studies are in AI scribe hallucination rates.
Who should score
Use clinicians who know the specialty and, where possible, did not conduct the consultation. Two raters score the same subset independently, then compare and agree a rule for each disagreement, which keeps scores consistent from week to week.
Keep language models out of the scoring role. In a Swiss proof-of-concept study, three models scoring notes with a modified PDQI-9 gave high marks, a median above 47 of 50, yet disagreed with each other systematically, with a Krippendorff's alpha of minus 0.433.4 The authors concluded that models cannot yet reliably replace human evaluation.4
How many notes to score
Scale the sample to the decision. For a first look at one note type, 30 notes scored on the PDQI-9 with a full error pass is a practical floor, and double-scoring 10 of them checks agreement. To compare two settings or two templates, score at least that many in each.
Count sentences as well as notes. At the 1.47% rate reported in the vendor-authored study, a sample of 1,000 sentences, roughly 30 to 40 notes, would be expected to contain around 15 hallucinated sentences, enough to see their pattern.2 When a sample shows no major errors, report it as zero in that many notes, since a small sample never proves a rate is zero.
Report every result with its denominator, as set out in how to report clinic AI pilot results.
Where the clinician's signature fits
With wai, the ambient scribe drafts the note during the consultation, structured the way the clinician writes, and the clinician reviews, edits and signs it.5
A month of scoring shows where to look first when reviewing, often the plan and any negated finding. The note checker looks at a single note, and our guide to AI scribe accuracy sets the wider evidence in context.
Where to start
First, pick one note type and collect 30 consecutive drafted notes with their recordings or transcripts. Second, agree the rubric below with a second clinician and double-score the first 10. Third, score the rest, with a full sentence-level error pass on each.
Fourth, tally major and minor errors by type and section, and the PDQI-9 total and item scores. Fifth, share the results with the clinicians using the tool, agree what to check first at sign-off, and repeat the exercise after any change of template or model, so every note is signed knowing exactly where to look.
Score a week of wai drafts against your own consultations, and see what changes once you know where to look.
Frequently asked questions
What is the PDQI-9?
A validated nine-item instrument for rating physician note quality, each item scored 1 to 5, giving a total out of 45.1
What is the difference between a hallucination and an omission?
A hallucination adds content the consultation does not support, and an omission leaves out something clinically relevant that was said, in the definitions of a vendor-authored framework.2
Which errors matter most?
Major ones, those that could change diagnosis or management. In a vendor-authored study of 12,999 sentences, 44% of hallucinated sentences were major, against 16.7% of omissions.2
Can an AI model score the notes for us?
Not reliably yet. Three models scoring the same notes disagreed systematically in a 2026 study.4 Use clinician raters.
How long does the review take?
Plan an afternoon a week for a month. Time per note depends on its length and on how practised the raters are, and having the transcript to hand speeds the error pass.
Who is responsible for the final note?
The clinician who signs it. wai drafts, and the clinician reviews, edits and signs.5
Sources
- 1Applied Clinical Informatics, Stetson and colleagues, 2012Assessing Electronic Note Quality Using the Physician Documentation Quality Instrument (PDQI-9)Read 14 October 2026Read source
- 2npj Digital Medicine, Asgari and colleagues, 2025A framework to assess clinical safety and hallucination rates of LLMs for medical text summarisation. Authors employed by a clinical documentation vendorRead 14 October 2026Read source
- 3Annals of Internal Medicine, Reddy and colleagues, 2026Rapid Evaluation of Artificial Intelligence Technology Used for Ambient Dictation in Primary Care, Comparing the Quality of Documentation of AI-Generated and Human-Produced Clinical NotesRead 14 October 2026Read source
- 4JMIR AI, Gładysz and colleagues, 2026AI-Assisted Medical Documentation in a Multilingual Swiss Health Care System, Proof-of-Concept StudyRead 14 October 2026Read source
- 5wai, 2026wai for cliniciansRead 14 October 2026Read source
Every note signed knowing exactly where to look.
Start a seven-day trial for one clinician, with limited use, or book fifteen minutes with a clinician who uses wai daily. We reply within one working day.
Partner with us
See it on your own patients, then build it with us
Thirty minutes on your own patients, doctors and rules, in your market. We reply within one working day, and the clinics that partner with us early shape what we build next.
See what wai does between the visits