65 lines
2.9 KiB
Markdown
65 lines
2.9 KiB
Markdown
# Dynamic question quality standard
|
|
|
|
Quality-standard-2 blueprints are eligible for learner runs. Older blueprints
|
|
remain stored only so signed questions and reports can be regenerated.
|
|
|
|
Quality-standard-3 blueprints add Elo-adaptive difficulty. Each active objective
|
|
must provide at least two families at every tier: foundation (850), standard
|
|
(1050), advanced (1250), and expert (1450). The service targets an item about 70
|
|
Elo below effective learner ability, approximately 60% expected correctness.
|
|
Effective ability begins as an equal blend of overall and category question-bank
|
|
Elo, then shifts toward dynamic objective Elo up to an 80% objective weight after
|
|
10 attempts. Aggregate outcomes calibrate each case family's Elo over time.
|
|
|
|
Quality-standard-4 blueprints add Step 3 coverage and lifecycle metadata. They
|
|
must declare the organ system, physician task, encounter frame, site of care,
|
|
tested concepts, medical reviewer, review date, and review expiration. Every
|
|
distractor must identify the misconception it represents. An expired medical
|
|
review prevents publication.
|
|
|
|
## Adaptive learning loop
|
|
|
|
New questions are chosen using a 50/30/20 policy:
|
|
|
|
- 50% scheduled concept reviews, when any are due.
|
|
- 30% underexposed blueprint areas, adjusted by `blueprintWeight`.
|
|
- 20% interleaved maintenance from the learner's weakest objectives.
|
|
|
|
Incorrect answers enter concept review; skips and correct first attempts do not.
|
|
Reviews use a freshly generated variant at 1, 3, 7, 21, and 45 days. An
|
|
incorrect review resets the interval to one day.
|
|
|
|
## Content requirements
|
|
|
|
- At least three curated case families per objective.
|
|
- One invariant `correctConceptId` across every family.
|
|
- A complete clinical vignette, focused lead-in, and coherent correlated data.
|
|
- At least six plausible distractors with an explanation for every answer.
|
|
- A complete explanation, key takeaway, USMLE pearl, review follow-up, and
|
|
wrong-answer remediation.
|
|
- At least one authoritative reference.
|
|
- Quality-standard-4 content includes misconception IDs and current medical
|
|
review metadata.
|
|
|
|
Promotion validates 10,000 seeds for every quality-standard-2 blueprint. Existing
|
|
blueprint versions are immutable; content changes require a version increment.
|
|
|
|
## Blinded acceptance
|
|
|
|
Create a balanced packet of three generated variants per gold objective and the
|
|
same number of authored questions:
|
|
|
|
```text
|
|
dotnet QuestionsApi.dll export-dynamic-review \
|
|
--source /app/content/dynamic_questions \
|
|
--output /tmp/dynamic-review
|
|
```
|
|
|
|
Keep `answer-key.json` separate from `review-packet.json`. Two clinical reviewers
|
|
score realism, distractor quality, explanation quality, and Step 3 relevance from
|
|
1-5 and predict whether each item is authored or dynamic.
|
|
|
|
The pilot passes when there are no critical medical or ambiguity findings, every
|
|
dimension averages at least 4/5, and source-identification accuracy is no greater
|
|
than 65%. Corrections are published as a new blueprint version and reviewed again.
|