Upgrade dynamic questions to authored parity

2026-07-23 14:11:18 -05:00
parent e174b5db15
commit 92c5688b55
12 changed files with 690 additions and 18 deletions
@@ -0,0 +1,36 @@
+# Dynamic question quality standard
+
+Quality-standard-2 blueprints are eligible for learner runs. Older blueprints
+remain stored only so signed questions and reports can be regenerated.
+
+## Content requirements
+
+- At least three curated case families per objective.
+- One invariant `correctConceptId` across every family.
+- A complete clinical vignette, focused lead-in, and coherent correlated data.
+- At least six plausible distractors with an explanation for every answer.
+- A complete explanation, key takeaway, USMLE pearl, review follow-up, and
+  wrong-answer remediation.
+- At least one authoritative reference.
+
+Promotion validates 10,000 seeds for every quality-standard-2 blueprint. Existing
+blueprint versions are immutable; content changes require a version increment.
+
+## Blinded acceptance
+
+Create a balanced packet of three generated variants per gold objective and the
+same number of authored questions:
+
+```text
+dotnet QuestionsApi.dll export-dynamic-review \
+  --source /app/content/dynamic_questions \
+  --output /tmp/dynamic-review
+```
+
+Keep `answer-key.json` separate from `review-packet.json`. Two clinical reviewers
+score realism, distractor quality, explanation quality, and Step 3 relevance from
+1–5 and predict whether each item is authored or dynamic.
+
+The pilot passes when there are no critical medical or ambiguity findings, every
+dimension averages at least 4/5, and source-identification accuracy is no greater
+than 65%. Corrections are published as a new blueprint version and reviewed again.