step3/QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md

Question Bank Distractor Enrichment Runbook

Use this process to expand existing questions beyond five choices. The goal is to retain every clinically useful distractor, not to reach a fixed pool size.

Preconditions

• Confirm step3_answer_statistics_v2_backfill_complete 1 on every API pod.
• Export the full bank through GET /api/questions/admin/export with a SUPER token held only in memory.
• Keep the export as the immutable baseline for post-write comparison.
• Do not change original choices, correct-answer IDs, stems, or explanations.

Authoring Pass

For each question, ask an authoring model to explore all natural misconception categories supported by the exact stem. Require every proposed distractor to include:

• a distinct, plausible answer at the same granularity as existing choices;
• an explanation of why it is wrong for this patient;
• a wrong-answer follow-up with exactly three unique A-C choices;
• a keyed follow-up fact stated explicitly or nearly explicitly in the stem;
• direct authoritative references for medical claims.

Do not specify a desired number. Zero candidates is valid when no additional answer can be made both plausible and unambiguously wrong.

Independent Audit

Use a separate model context to evaluate every candidate individually. Approve only candidates that are deceptive to a Step 3 learner, materially distinct, and unambiguously wrong under the exact vignette. Reject remote or absurd actions, padding, duplicate concepts, contrived sequencing language, conditionally correct answers, unsupported claims, and follow-ups whose keyed fact does not specifically defeat the distractor.

The auditor should identify missing natural misconception categories without imposing a numerical quota. Run focused generation and an independent audit for those gaps. If a requested category cannot produce an unambiguously wrong option, record a quality waiver instead of forcing it into the bank.

Merge Validation

Append approved candidates and assign stable choice IDs sequentially after the existing IDs. For each question, verify:

• 5-20 unique choice IDs and case-insensitively unique choice text;
• the original correct-answer ID and original choices are unchanged;
• choiceExplanations has exactly one entry per choice;
• wrongAnswerFollowups has exactly one entry per incorrect choice and none for the correct choice;
• every follow-up has exactly A-C choices and a valid keyed answer;
• new choiceSelectionPercentages entries start at zero;
• authoritative sources supporting approved additions are merged into the question's references array without duplicating existing URLs;
• no unresolved coverage issue remains unless documented as a waiver.

Deserialize the payload as BulkUpsertQuestionsRequest and run the actual server QuestionValidator.Validate method against every question before upload.

Upload And Verification

Split validated JSON with a UTF-8-aware tool such as Node. Do not parse a UTF-8-without-BOM payload using Windows PowerShell Get-Content; it can silently corrupt non-ASCII clinical text. Upload exact batch files with Content-Type: application/json; charset=utf-8.

After upload:

1. Export the complete production bank again.
2. Compare stored choices, correct IDs, explanations, and follow-ups exactly.
3. Compare every original correct ID and choice list with the baseline.
4. Confirm pool-aware search returns full pools with the correct answer.
5. Confirm legacy search returns exactly five choices with the correct answer.
6. Remove temporary prompts, exports, tokens, batches, and model output.

2026-07-13 Run

• Questions reviewed: 349
• Additional distractors approved: 1,113
• Final choices: 2,857
• Final pool range: 5-17 choices
• Questions retaining no additional distractor after audit: 28
• Focused coverage repairs approved: 16
• Quality waivers: 3
• Server validator: 349/349 passed
• Production exact comparison: 349/349 passed
• Live pool-aware sample: 38/40 had more than five choices; all included the correct answer
• Live legacy sample: 40/40 returned five choices and included the correct answer

The quality waivers covered severe hypercalcemia antiresorptive therapy, residual gastric-cancer risk after H pylori eradication, and unspecified oral antifungal toxicity. In each case, forcing the category would have created a co-correct or otherwise ambiguous option.