From 0fe47de423919618af4ce2cc2141451974e7c9d0 Mon Sep 17 00:00:00 2001 From: hz Date: Mon, 13 Jul 2026 18:18:38 -0500 Subject: [PATCH] Document distractor enrichment workflow --- QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md | 95 ++++++++++++++++++++++++++ README.md | 4 ++ 2 files changed, 99 insertions(+) create mode 100644 QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md diff --git a/QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md b/QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md new file mode 100644 index 0000000..91b7d94 --- /dev/null +++ b/QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md @@ -0,0 +1,95 @@ +# Question Bank Distractor Enrichment Runbook + +Use this process to expand existing questions beyond five choices. The goal is +to retain every clinically useful distractor, not to reach a fixed pool size. + +## Preconditions + +- Confirm `step3_answer_statistics_v2_backfill_complete 1` on every API pod. +- Export the full bank through `GET /api/questions/admin/export` with a SUPER + token held only in memory. +- Keep the export as the immutable baseline for post-write comparison. +- Do not change original choices, correct-answer IDs, stems, or explanations. + +## Authoring Pass + +For each question, ask an authoring model to explore all natural misconception +categories supported by the exact stem. Require every proposed distractor to +include: + +- a distinct, plausible answer at the same granularity as existing choices; +- an explanation of why it is wrong for this patient; +- a wrong-answer follow-up with exactly three unique A-C choices; +- a keyed follow-up fact stated explicitly or nearly explicitly in the stem; +- direct authoritative references for medical claims. + +Do not specify a desired number. Zero candidates is valid when no additional +answer can be made both plausible and unambiguously wrong. + +## Independent Audit + +Use a separate model context to evaluate every candidate individually. Approve +only candidates that are deceptive to a Step 3 learner, materially distinct, +and unambiguously wrong under the exact vignette. Reject remote or absurd +actions, padding, duplicate concepts, contrived sequencing language, +conditionally correct answers, unsupported claims, and follow-ups whose keyed +fact does not specifically defeat the distractor. + +The auditor should identify missing natural misconception categories without +imposing a numerical quota. Run focused generation and an independent audit for +those gaps. If a requested category cannot produce an unambiguously wrong +option, record a quality waiver instead of forcing it into the bank. + +## Merge Validation + +Append approved candidates and assign stable choice IDs sequentially after the +existing IDs. For each question, verify: + +- 5-20 unique choice IDs and case-insensitively unique choice text; +- the original correct-answer ID and original choices are unchanged; +- `choiceExplanations` has exactly one entry per choice; +- `wrongAnswerFollowups` has exactly one entry per incorrect choice and none + for the correct choice; +- every follow-up has exactly A-C choices and a valid keyed answer; +- new `choiceSelectionPercentages` entries start at zero; +- no unresolved coverage issue remains unless documented as a waiver. + +Deserialize the payload as `BulkUpsertQuestionsRequest` and run the actual +server `QuestionValidator.Validate` method against every question before upload. + +## Upload And Verification + +Split validated JSON with a UTF-8-aware tool such as Node. Do not parse a +UTF-8-without-BOM payload using Windows PowerShell `Get-Content`; it can silently +corrupt non-ASCII clinical text. Upload exact batch files with +`Content-Type: application/json; charset=utf-8`. + +After upload: + +1. Export the complete production bank again. +2. Compare stored choices, correct IDs, explanations, and follow-ups exactly. +3. Compare every original correct ID and choice list with the baseline. +4. Confirm pool-aware search returns full pools with the correct answer. +5. Confirm legacy search returns exactly five choices with the correct answer. +6. Remove temporary prompts, exports, tokens, batches, and model output. + +## 2026-07-13 Run + +- Questions reviewed: 349 +- Additional distractors approved: 1,113 +- Final choices: 2,857 +- Final pool range: 5-17 choices +- Questions retaining no additional distractor after audit: 28 +- Focused coverage repairs approved: 16 +- Quality waivers: 3 +- Server validator: 349/349 passed +- Production exact comparison: 349/349 passed +- Live pool-aware sample: 38/40 had more than five choices; all included the + correct answer +- Live legacy sample: 40/40 returned five choices and included the correct + answer + +The quality waivers covered severe hypercalcemia antiresorptive therapy, +residual gastric-cancer risk after *H pylori* eradication, and unspecified oral +antifungal toxicity. In each case, forcing the category would have created a +co-correct or otherwise ambiguous option. diff --git a/README.md b/README.md index 6d8b06c..547e919 100644 --- a/README.md +++ b/README.md @@ -124,6 +124,10 @@ distractors: record an exception instead of forcing a weak or co-correct answer. Do not expand production pools until the v2 exposure-statistics backfill metric reports completion. +See [QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md](QBANK_DISTRACTOR_ENRICHMENT_RUNBOOK.md) +for the repeatable authoring, independent-audit, merge, and production +verification procedure. + ## Duplicate Question Review Every question must be checked for duplicates before publication. This review