How it works
Checking what the evidence says against what the answer said
Step 1
Extract material points at ingest
Every source that enters the library is read once, in full, when it arrives. We extract the points a clinician would need to know before acting, record the type and severity of each, and keep a pointer to the exact passage.
Doing this at ingest means a question only has to match against points that have already been extracted. The library becomes a structured layer of what each source says matters, not just a pile of text to search.
Point types
- Contraindications
- Dose limits
- Monitoring requirements
- Drug interactions
- Alternatives
- Red flags
- Population caveats
- Timing
- Retracted or superseded findings
Severity
Mild, moderate or severe, following the NOHARM severity scale.
Step 2
Decide what applies to this question
Not every point in every source is relevant. A contraindication for the drug being asked about is material. A caveat about a different patient population is not. For each question we weigh which points apply before checking anything, because this step decides whether the verifier is precise or just noisy.
Step 3
Check each point against the answer
For each material point, we ask whether the answer states it. Two models from different families judge every point independently. A point counts as missing only when both agree it is missing, and every disagreement is kept as a test case.
The result is a completeness score weighted by severity. If any severe point is missing, the answer is not marked verified.
Pass 1 · Answer → source
A citation check: is each sentence in the answer supported by a source? Here, all three are.
Pass 2 · Source → answer
The omission check: is each material point in the sources stated in the answer? One has nowhere to land.
Step 4
Show the clinician what is missing
The answer streams as it does now. The completeness check follows it, listing each missing point under the heading “Not stated in this answer”, with its severity and a link to the source passage.
Return the completeness check within one second at the 95th percentile, after the answer has streamed.
Source: Omission verifier design note, verifier latency budget
Limits
What it can't do
It can only check against evidence it has
If the right source was never retrieved, the verifier cannot know what it said. Retrieval quality sets the ceiling, which is why we check against a wider set of sources than the answer was written from.
Abstracts leave things out too
Dose caps and contraindications often appear only in full text, drug labels and guidelines. We add drug labels and guideline summaries as references alongside the literature.
Too many flags is its own failure
A badge that fires on every answer gets ignored. The false-flag rate on complete answers is a release gate, not an afterthought.
It reviews; it doesn't decide
The verifier tells a clinician what an answer did not state. What to do about it stays with the clinician.
Design-partner pilots
Help us build it on your cases
We are looking for a small number of hospitals, medical centers, clinicians and research groups to shape the verifier and its evaluation with us.
- ≥85%Target
of severe omissions caught
- ≤10%Target
of complete answers wrongly flagged