step3/docs/dynamic-question-quality.md

Dynamic question quality standard

Quality-standard-2 blueprints are eligible for learner runs. Older blueprints remain stored only so signed questions and reports can be regenerated.

Content requirements

• At least three curated case families per objective.
• One invariant correctConceptId across every family.
• A complete clinical vignette, focused lead-in, and coherent correlated data.
• At least six plausible distractors with an explanation for every answer.
• A complete explanation, key takeaway, USMLE pearl, review follow-up, and wrong-answer remediation.
• At least one authoritative reference.

Promotion validates 10,000 seeds for every quality-standard-2 blueprint. Existing blueprint versions are immutable; content changes require a version increment.

Blinded acceptance

Create a balanced packet of three generated variants per gold objective and the same number of authored questions:

dotnet QuestionsApi.dll export-dynamic-review \
  --source /app/content/dynamic_questions \
  --output /tmp/dynamic-review

Keep answer-key.json separate from review-packet.json. Two clinical reviewers score realism, distractor quality, explanation quality, and Step 3 relevance from 1–5 and predict whether each item is authored or dynamic.

The pilot passes when there are no critical medical or ambiguity findings, every dimension averages at least 4/5, and source-identification accuracy is no greater than 65%. Corrections are published as a new blueprint version and reviewed again.