Document distractor enrichment workflow

QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md

@@ -0,0 +1,95 @@
+# Question Bank Distractor Enrichment Runbook
+
+Use this process to expand existing questions beyond five choices. The goal is
+to retain every clinically useful distractor, not to reach a fixed pool size.
+
+## Preconditions
+
+- Confirm `step3_answer_statistics_v2_backfill_complete 1` on every API pod.
+- Export the full bank through `GET /api/questions/admin/export` with a SUPER
+  token held only in memory.
+- Keep the export as the immutable baseline for post-write comparison.
+- Do not change original choices, correct-answer IDs, stems, or explanations.
+
+## Authoring Pass
+
+For each question, ask an authoring model to explore all natural misconception
+categories supported by the exact stem. Require every proposed distractor to
+include:
+
+- a distinct, plausible answer at the same granularity as existing choices;
+- an explanation of why it is wrong for this patient;
+- a wrong-answer follow-up with exactly three unique A-C choices;
+- a keyed follow-up fact stated explicitly or nearly explicitly in the stem;
+- direct authoritative references for medical claims.
+
+Do not specify a desired number. Zero candidates is valid when no additional
+answer can be made both plausible and unambiguously wrong.
+
+## Independent Audit
+
+Use a separate model context to evaluate every candidate individually. Approve
+only candidates that are deceptive to a Step 3 learner, materially distinct,
+and unambiguously wrong under the exact vignette. Reject remote or absurd
+actions, padding, duplicate concepts, contrived sequencing language,
+conditionally correct answers, unsupported claims, and follow-ups whose keyed
+fact does not specifically defeat the distractor.
+
+The auditor should identify missing natural misconception categories without
+imposing a numerical quota. Run focused generation and an independent audit for
+those gaps. If a requested category cannot produce an unambiguously wrong
+option, record a quality waiver instead of forcing it into the bank.
+
+## Merge Validation
+
+Append approved candidates and assign stable choice IDs sequentially after the
+existing IDs. For each question, verify:
+
+- 5-20 unique choice IDs and case-insensitively unique choice text;
+- the original correct-answer ID and original choices are unchanged;
+- `choiceExplanations` has exactly one entry per choice;
+- `wrongAnswerFollowups` has exactly one entry per incorrect choice and none
+  for the correct choice;
+- every follow-up has exactly A-C choices and a valid keyed answer;
+- new `choiceSelectionPercentages` entries start at zero;
+- no unresolved coverage issue remains unless documented as a waiver.
+
+Deserialize the payload as `BulkUpsertQuestionsRequest` and run the actual
+server `QuestionValidator.Validate` method against every question before upload.
+
+## Upload And Verification
+
+Split validated JSON with a UTF-8-aware tool such as Node. Do not parse a
+UTF-8-without-BOM payload using Windows PowerShell `Get-Content`; it can silently
+corrupt non-ASCII clinical text. Upload exact batch files with
+`Content-Type: application/json; charset=utf-8`.
+
+After upload:
+
+1. Export the complete production bank again.
+2. Compare stored choices, correct IDs, explanations, and follow-ups exactly.
+3. Compare every original correct ID and choice list with the baseline.
+4. Confirm pool-aware search returns full pools with the correct answer.
+5. Confirm legacy search returns exactly five choices with the correct answer.
+6. Remove temporary prompts, exports, tokens, batches, and model output.
+
+## 2026-07-13 Run
+
+- Questions reviewed: 349
+- Additional distractors approved: 1,113
+- Final choices: 2,857
+- Final pool range: 5-17 choices
+- Questions retaining no additional distractor after audit: 28
+- Focused coverage repairs approved: 16
+- Quality waivers: 3
+- Server validator: 349/349 passed
+- Production exact comparison: 349/349 passed
+- Live pool-aware sample: 38/40 had more than five choices; all included the
+  correct answer
+- Live legacy sample: 40/40 returned five choices and included the correct
+  answer
+
+The quality waivers covered severe hypercalcemia antiresorptive therapy,
+residual gastric-cancer risk after *H pylori* eradication, and unspecified oral
+antifungal toxicity. In each case, forcing the category would have created a
+co-correct or otherwise ambiguous option.

README.md

@@ -124,6 +124,10 @@ distractors: record an exception instead of forcing a weak or co-correct
 answer. Do not expand production pools until the v2 exposure-statistics
 backfill metric reports completion.
 
+See [QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md](QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md)
+for the repeatable authoring, independent-audit, merge, and production
+verification procedure.
+
 ## Duplicate Question Review
 
 Every question must be checked for duplicates before publication. This review