step3/docs/dynamic-question-quality.md

Dynamic question quality standard

Quality-standard-2 blueprints are eligible for learner runs. Older blueprints remain stored only so signed questions and reports can be regenerated.

Quality-standard-3 blueprints add Elo-adaptive difficulty. Each active objective must provide at least two families at every tier: foundation (850), standard (1050), advanced (1250), and expert (1450). The service targets an item about 70 Elo below effective learner ability, approximately 60% expected correctness. Effective ability begins as an equal blend of overall and category question-bank Elo, then shifts toward dynamic objective Elo up to an 80% objective weight after 10 attempts. Aggregate outcomes calibrate each case family's Elo over time.

Quality-standard-4 blueprints add Step 3 coverage and lifecycle metadata. They must declare the organ system, physician task, encounter frame, site of care, tested concepts, medical reviewer, review date, and review expiration. Every distractor must identify the misconception it represents. An expired medical review prevents publication.

Adaptive learning loop

New questions are chosen using a 50/30/20 policy:

• 50% scheduled concept reviews, when any are due.
• 30% underexposed blueprint areas, adjusted by blueprintWeight.
• 20% interleaved maintenance from the learner's weakest objectives.

Incorrect answers enter concept review; skips and correct first attempts do not. Reviews use a freshly generated variant at 1, 3, 7, 21, and 45 days. An incorrect review resets the interval to one day.

Content requirements

• At least three curated case families per objective.
• One invariant correctConceptId across every family.
• A complete clinical vignette, focused lead-in, and coherent correlated data.
• At least six plausible distractors with an explanation for every answer.
• A complete explanation, key takeaway, USMLE pearl, review follow-up, and wrong-answer remediation.
• At least one authoritative reference.
• Quality-standard-4 content includes misconception IDs and current medical review metadata.

Promotion validates 10,000 seeds for every quality-standard-2 blueprint. Existing blueprint versions are immutable; content changes require a version increment.

Blinded acceptance

Create a balanced packet of three generated variants per gold objective and the same number of authored questions:

dotnet QuestionsApi.dll export-dynamic-review \
  --source /app/content/dynamic_questions \
  --output /tmp/dynamic-review

Keep answer-key.json separate from review-packet.json. Two clinical reviewers score realism, distractor quality, explanation quality, and Step 3 relevance from 1-5 and predict whether each item is authored or dynamic.

The pilot passes when there are no critical medical or ambiguity findings, every dimension averages at least 4/5, and source-identification accuracy is no greater than 65%. Corrections are published as a new blueprint version and reviewed again.