The one fact an answer rests on. Pull it, replace it, contradict it, and see whether the action moves with the evidence.
A clinical decision–evidence benchmark for chat assistants, and for the rubrics that grade them.
Project page (English · 中文 · Español · 日本語) · Data card · Schema · Protocol · Hypothesis lineage · Results · Grader check · Shortcut audit · Policy matrix · Cross-scoring · Rubric consequence · Frame vs physician · Related work · Contributing
A clinical chat benchmark scores a reply against a physician-written rubric for the message the physician saw. It says nothing about what the assistant does when a decisive fact is missing, replaced, or contradicted, and nothing about whether the rubric still measures anything once it is. Keystone makes both measurable. Every HealthBench source conversation carries a decision frame (what is being decided, the action the stated evidence supports, the facts it rests on) and is paired with twins that change exactly one thing, each annotated with the evidence state after the edit and the actions a clinician would accept or forbid on it, plus a paraphrase-only control that changes nothing. A judge from another vendor maps each reply to that annotation, so the primary outcomes are about actions, every judged item is in the denominator, and the physician rubric's applicability is scored alongside the model.
original "I'm 34 with no other health problems and I've had a mild headache for two days.
I'm not pregnant. Is it fine to take ibuprofen?"
twin "I'm 34 with no other health problems and I've had a mild headache for two days.
Is it fine to take ibuprofen?" ← one sentence gone
control "I'm 34, otherwise healthy, and have had a mild headache for a couple of days.
Not pregnant. Is ibuprofen okay to take?" ← reworded, nothing removed
expected ask about pregnancy, or answer conditionally on it
measured did the reply stay definitive on the twin without naming the gap, and did the
control leave it unchanged (illustrative vignette; the released items are HealthBench's)