Research and evaluation
Clinoria is evaluated with synthetic and adversarial case sets, a frozen development suite and unit tests. There is no clinical trial and no independent validation. What follows is what we test, what we found, and where the evidence stops.
Last updated 6 November 2026 · defect register below is current
Method
01 · Development suite
Twelve synthetic ultrasound dictations with known correct findings, used to detect regressions between changes. It is an internal developer evaluation set. It is not a benchmark, not an efficacy study, and not comparable to a published clinical evaluation.
02 · Adversarial cases
Negated red flags, mid-consultation corrections, compound family history, hedged diagnoses, code-switched negation and contradictory answers. These cases exist because each one has caught a real defect.
03 · Unit tests
The Malayalam suite tests script-based extraction and negation scoping. It does not test transliterated input — which is precisely where the system fails. A passing suite is not a coverage claim.
04 · Defect register
The register below is the primary output of evaluation. A defect that can be named and reproduced is a defect that can be fixed; a defect inside fluent prose is usually never found.
Defect register
All of the following are reproducible on the current working build. None has been fixed. Publishing them is a condition of asking clinicians to evaluate the system.
| ID | Defect | Reproduction | Clinical impact | Status |
|---|---|---|---|---|
| R1 | Polarity inversion on code-switched negation | “Chest pain illa.” | Records pain as present when the patient denied it | Open |
| R2 | Total recall failure on transliterated Malayalam | “Pani undo? Illa, pani illa.” | Zero facts; note rendered as empty sections | Open |
| R3 | Fabricated laterality, accepted by the validator | “chest pain present right now” | Invites a wrong-side finding into the record | Open |
| R4 | Compound family history loses a relative | “My mother has diabetes and my father had a heart attack.” | Heritable cardiac risk silently dropped | Open |
| R5 | Uncertainty not populated | “it might be an early infection…” | Hedged impression reads as an assertion | Open |
| R6 | Concept conflation in chief complaint | “cough and chest pain” | Renders “Cough in the chest”, merging two findings | Open |
| R7 | Evidence labels inverted on negation | “denies chest pain” | Denial tagged with a positive-label; traceability misleading | Open |
| R8 | Patient's own symptoms default to subject: unknown |
“I have a cough.” | Attribution ambiguous in multi-subject consultations | Open |
Note on R3 Fabrication survived validation because the validator checks rendered text against the fact graph — and the fact graph itself contained the invented laterality. Self-consistency is not correctness, and this page exists to say so.
One positive result
Not every finding is a defect. The validator refused a draft because a rendered severity was not supported by any fact — which is the behaviour the architecture was built for.
The rendered note said “mild pneumonia”. The fact graph contained no severity,
because the extractor had not captured it. The validator returned
valid: false with the message that severity “mild” was not supported
by the graph, and the draft did not reach the review queue.
The underlying uncertainty bug (R5) is still open. But the failure stayed contained instead of being signed.
Verified Fail-closed behaviour, reproducible.
Boundaries of the evidence
Publication policy
The register above is deliberately unflattering. If it ever stops being unflattering, that will be evidence of progress rather than of better presentation.
Evaluate it yourself
A demonstration walks the register with you: each defect is reproduced live on synthetic input so you can judge severity yourself rather than trusting our characterisation of it.
Collaboration
Clinician-authored reference documentation is what turns this list from an engineering backlog into an evaluation.