Document distractor enrichment workflow

2026-07-13 18:18:38 -05:00
parent 361847cae2
commit 0fe47de423
2 changed files with 99 additions and 0 deletions
@@ -0,0 +1,95 @@
 # Question Bank Distractor Enrichment Runbook
 Use this process to expand existing questions beyond five choices. The goal is
 to retain every clinically useful distractor, not to reach a fixed pool size.
 ## Preconditions
 - Confirm `step3_answer_statistics_v2_backfill_complete 1` on every API pod.
 - Export the full bank through `GET /api/questions/admin/export` with a SUPER
  token held only in memory.
 - Keep the export as the immutable baseline for post-write comparison.
 - Do not change original choices, correct-answer IDs, stems, or explanations.
 ## Authoring Pass
 For each question, ask an authoring model to explore all natural misconception
 categories supported by the exact stem. Require every proposed distractor to
 include:
 - a distinct, plausible answer at the same granularity as existing choices;
 - an explanation of why it is wrong for this patient;
 - a wrong-answer follow-up with exactly three unique A-C choices;
 - a keyed follow-up fact stated explicitly or nearly explicitly in the stem;
 - direct authoritative references for medical claims.
 Do not specify a desired number. Zero candidates is valid when no additional
 answer can be made both plausible and unambiguously wrong.
 ## Independent Audit
 Use a separate model context to evaluate every candidate individually. Approve
 only candidates that are deceptive to a Step 3 learner, materially distinct,
 and unambiguously wrong under the exact vignette. Reject remote or absurd
 actions, padding, duplicate concepts, contrived sequencing language,
 conditionally correct answers, unsupported claims, and follow-ups whose keyed
 fact does not specifically defeat the distractor.
 The auditor should identify missing natural misconception categories without
 imposing a numerical quota. Run focused generation and an independent audit for
 those gaps. If a requested category cannot produce an unambiguously wrong
 option, record a quality waiver instead of forcing it into the bank.
 ## Merge Validation
 Append approved candidates and assign stable choice IDs sequentially after the
 existing IDs. For each question, verify:
 - 5-20 unique choice IDs and case-insensitively unique choice text;
 - the original correct-answer ID and original choices are unchanged;
 - `choiceExplanations` has exactly one entry per choice;
 - `wrongAnswerFollowups` has exactly one entry per incorrect choice and none
  for the correct choice;
 - every follow-up has exactly A-C choices and a valid keyed answer;
 - new `choiceSelectionPercentages` entries start at zero;
 - no unresolved coverage issue remains unless documented as a waiver.
 Deserialize the payload as `BulkUpsertQuestionsRequest` and run the actual
 server `QuestionValidator.Validate` method against every question before upload.
 ## Upload And Verification
 Split validated JSON with a UTF-8-aware tool such as Node. Do not parse a
 UTF-8-without-BOM payload using Windows PowerShell `Get-Content`; it can silently
 corrupt non-ASCII clinical text. Upload exact batch files with
 `Content-Type: application/json; charset=utf-8`.
 After upload:
 1. Export the complete production bank again.
 2. Compare stored choices, correct IDs, explanations, and follow-ups exactly.
 3. Compare every original correct ID and choice list with the baseline.
 4. Confirm pool-aware search returns full pools with the correct answer.
 5. Confirm legacy search returns exactly five choices with the correct answer.
 6. Remove temporary prompts, exports, tokens, batches, and model output.
 ## 2026-07-13 Run
 - Questions reviewed: 349
 - Additional distractors approved: 1,113
 - Final choices: 2,857
 - Final pool range: 5-17 choices
 - Questions retaining no additional distractor after audit: 28
 - Focused coverage repairs approved: 16
 - Quality waivers: 3
 - Server validator: 349/349 passed
 - Production exact comparison: 349/349 passed
 - Live pool-aware sample: 38/40 had more than five choices; all included the
  correct answer
 - Live legacy sample: 40/40 returned five choices and included the correct
  answer
 The quality waivers covered severe hypercalcemia antiresorptive therapy,
 residual gastric-cancer risk after *H pylori* eradication, and unspecified oral
 antifungal toxicity. In each case, forcing the category would have created a
 co-correct or otherwise ambiguous option.
@@ -124,6 +124,10 @@ distractors: record an exception instead of forcing a weak or co-correct
 answer. Do not expand production pools until the v2 exposure-statistics
 backfill metric reports completion.
 See [QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md](QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md)
 for the repeatable authoring, independent-audit, merge, and production
 verification procedure.
 ## Duplicate Question Review
 Every question must be checked for duplicates before publication. This review