98 lines
4.3 KiB
Markdown
98 lines
4.3 KiB
Markdown
# Question Bank Distractor Enrichment Runbook
|
|
|
|
Use this process to expand existing questions beyond five choices. The goal is
|
|
to retain every clinically useful distractor, not to reach a fixed pool size.
|
|
|
|
## Preconditions
|
|
|
|
- Confirm `step3_answer_statistics_v2_backfill_complete 1` on every API pod.
|
|
- Export the full bank through `GET /api/questions/admin/export` with a SUPER
|
|
token held only in memory.
|
|
- Keep the export as the immutable baseline for post-write comparison.
|
|
- Do not change original choices, correct-answer IDs, stems, or explanations.
|
|
|
|
## Authoring Pass
|
|
|
|
For each question, ask an authoring model to explore all natural misconception
|
|
categories supported by the exact stem. Require every proposed distractor to
|
|
include:
|
|
|
|
- a distinct, plausible answer at the same granularity as existing choices;
|
|
- an explanation of why it is wrong for this patient;
|
|
- a wrong-answer follow-up with exactly three unique A-C choices;
|
|
- a keyed follow-up fact stated explicitly or nearly explicitly in the stem;
|
|
- direct authoritative references for medical claims.
|
|
|
|
Do not specify a desired number. Zero candidates is valid when no additional
|
|
answer can be made both plausible and unambiguously wrong.
|
|
|
|
## Independent Audit
|
|
|
|
Use a separate model context to evaluate every candidate individually. Approve
|
|
only candidates that are deceptive to a Step 3 learner, materially distinct,
|
|
and unambiguously wrong under the exact vignette. Reject remote or absurd
|
|
actions, padding, duplicate concepts, contrived sequencing language,
|
|
conditionally correct answers, unsupported claims, and follow-ups whose keyed
|
|
fact does not specifically defeat the distractor.
|
|
|
|
The auditor should identify missing natural misconception categories without
|
|
imposing a numerical quota. Run focused generation and an independent audit for
|
|
those gaps. If a requested category cannot produce an unambiguously wrong
|
|
option, record a quality waiver instead of forcing it into the bank.
|
|
|
|
## Merge Validation
|
|
|
|
Append approved candidates and assign stable choice IDs sequentially after the
|
|
existing IDs. For each question, verify:
|
|
|
|
- 5-20 unique choice IDs and case-insensitively unique choice text;
|
|
- the original correct-answer ID and original choices are unchanged;
|
|
- `choiceExplanations` has exactly one entry per choice;
|
|
- `wrongAnswerFollowups` has exactly one entry per incorrect choice and none
|
|
for the correct choice;
|
|
- every follow-up has exactly A-C choices and a valid keyed answer;
|
|
- new `choiceSelectionPercentages` entries start at zero;
|
|
- authoritative sources supporting approved additions are merged into the
|
|
question's `references` array without duplicating existing URLs;
|
|
- no unresolved coverage issue remains unless documented as a waiver.
|
|
|
|
Deserialize the payload as `BulkUpsertQuestionsRequest` and run the actual
|
|
server `QuestionValidator.Validate` method against every question before upload.
|
|
|
|
## Upload And Verification
|
|
|
|
Split validated JSON with a UTF-8-aware tool such as Node. Do not parse a
|
|
UTF-8-without-BOM payload using Windows PowerShell `Get-Content`; it can silently
|
|
corrupt non-ASCII clinical text. Upload exact batch files with
|
|
`Content-Type: application/json; charset=utf-8`.
|
|
|
|
After upload:
|
|
|
|
1. Export the complete production bank again.
|
|
2. Compare stored choices, correct IDs, explanations, and follow-ups exactly.
|
|
3. Compare every original correct ID and choice list with the baseline.
|
|
4. Confirm pool-aware search returns full pools with the correct answer.
|
|
5. Confirm legacy search returns exactly five choices with the correct answer.
|
|
6. Remove temporary prompts, exports, tokens, batches, and model output.
|
|
|
|
## 2026-07-13 Run
|
|
|
|
- Questions reviewed: 349
|
|
- Additional distractors approved: 1,113
|
|
- Final choices: 2,857
|
|
- Final pool range: 5-17 choices
|
|
- Questions retaining no additional distractor after audit: 28
|
|
- Focused coverage repairs approved: 16
|
|
- Quality waivers: 3
|
|
- Server validator: 349/349 passed
|
|
- Production exact comparison: 349/349 passed
|
|
- Live pool-aware sample: 38/40 had more than five choices; all included the
|
|
correct answer
|
|
- Live legacy sample: 40/40 returned five choices and included the correct
|
|
answer
|
|
|
|
The quality waivers covered severe hypercalcemia antiresorptive therapy,
|
|
residual gastric-cancer risk after *H pylori* eradication, and unspecified oral
|
|
antifungal toxicity. In each case, forcing the category would have created a
|
|
co-correct or otherwise ambiguous option.
|