45 lines
2.0 KiB
Markdown
45 lines
2.0 KiB
Markdown
# Dynamic question quality standard
|
||
|
||
Quality-standard-2 blueprints are eligible for learner runs. Older blueprints
|
||
remain stored only so signed questions and reports can be regenerated.
|
||
|
||
Quality-standard-3 blueprints add Elo-adaptive difficulty. Each active objective
|
||
must provide at least two families at every tier: foundation (850), standard
|
||
(1050), advanced (1250), and expert (1450). The service targets an item about 70
|
||
Elo below effective learner ability, approximately 60% expected correctness.
|
||
Effective ability begins as an equal blend of overall and category question-bank
|
||
Elo, then shifts toward dynamic objective Elo up to an 80% objective weight after
|
||
10 attempts. Aggregate outcomes calibrate each case family's Elo over time.
|
||
|
||
## Content requirements
|
||
|
||
- At least three curated case families per objective.
|
||
- One invariant `correctConceptId` across every family.
|
||
- A complete clinical vignette, focused lead-in, and coherent correlated data.
|
||
- At least six plausible distractors with an explanation for every answer.
|
||
- A complete explanation, key takeaway, USMLE pearl, review follow-up, and
|
||
wrong-answer remediation.
|
||
- At least one authoritative reference.
|
||
|
||
Promotion validates 10,000 seeds for every quality-standard-2 blueprint. Existing
|
||
blueprint versions are immutable; content changes require a version increment.
|
||
|
||
## Blinded acceptance
|
||
|
||
Create a balanced packet of three generated variants per gold objective and the
|
||
same number of authored questions:
|
||
|
||
```text
|
||
dotnet QuestionsApi.dll export-dynamic-review \
|
||
--source /app/content/dynamic_questions \
|
||
--output /tmp/dynamic-review
|
||
```
|
||
|
||
Keep `answer-key.json` separate from `review-packet.json`. Two clinical reviewers
|
||
score realism, distractor quality, explanation quality, and Step 3 relevance from
|
||
1–5 and predict whether each item is authored or dynamic.
|
||
|
||
The pilot passes when there are no critical medical or ambiguity findings, every
|
||
dimension averages at least 4/5, and source-identification accuracy is no greater
|
||
than 65%. Corrections are published as a new blueprint version and reviewed again.
|