# Question Bank Distractor Enrichment Runbook

Use this process to expand existing questions beyond five choices. The goal is
to retain every clinically useful distractor, not to reach a fixed pool size.

## Preconditions

- Confirm `step3_answer_statistics_v2_backfill_complete 1` on every API pod.
- Export the full bank through `GET /api/questions/admin/export` with a SUPER
  token held only in memory.
- Keep the export as the immutable baseline for post-write comparison.
- Do not change original choices, correct-answer IDs, stems, or explanations.

## Authoring Pass

For each question, ask an authoring model to explore all natural misconception
categories supported by the exact stem. Require every proposed distractor to
include:

- a distinct, plausible answer at the same granularity as existing choices;
- an explanation of why it is wrong for this patient;
- a wrong-answer follow-up with exactly three unique A-C choices;
- a keyed follow-up fact stated explicitly or nearly explicitly in the stem;
- direct authoritative references for medical claims.

Do not specify a desired number. Zero candidates is valid when no additional
answer can be made both plausible and unambiguously wrong.

## Independent Audit

Use a separate model context to evaluate every candidate individually. Approve
only candidates that are deceptive to a Step 3 learner, materially distinct,
and unambiguously wrong under the exact vignette. Reject remote or absurd
actions, padding, duplicate concepts, contrived sequencing language,
conditionally correct answers, unsupported claims, and follow-ups whose keyed
fact does not specifically defeat the distractor.

The auditor should identify missing natural misconception categories without
imposing a numerical quota. Run focused generation and an independent audit for
those gaps. If a requested category cannot produce an unambiguously wrong
option, record a quality waiver instead of forcing it into the bank.

## Merge Validation

Append approved candidates and assign stable choice IDs sequentially after the
existing IDs. For each question, verify:

- 5-20 unique choice IDs and case-insensitively unique choice text;
- the original correct-answer ID and original choices are unchanged;
- `choiceExplanations` has exactly one entry per choice;
- `wrongAnswerFollowups` has exactly one entry per incorrect choice and none
  for the correct choice;
- every follow-up has exactly A-C choices and a valid keyed answer;
- new `choiceSelectionPercentages` entries start at zero;
- authoritative sources supporting approved additions are merged into the
  question's `references` array without duplicating existing URLs;
- no unresolved coverage issue remains unless documented as a waiver.

Deserialize the payload as `BulkUpsertQuestionsRequest` and run the actual
server `QuestionValidator.Validate` method against every question before upload.

## Upload And Verification

Split validated JSON with a UTF-8-aware tool such as Node. Do not parse a
UTF-8-without-BOM payload using Windows PowerShell `Get-Content`; it can silently
corrupt non-ASCII clinical text. Upload exact batch files with
`Content-Type: application/json; charset=utf-8`.

After upload:

1. Export the complete production bank again.
2. Compare stored choices, correct IDs, explanations, and follow-ups exactly.
3. Compare every original correct ID and choice list with the baseline.
4. Confirm pool-aware search returns full pools with the correct answer.
5. Confirm legacy search returns exactly five choices with the correct answer.
6. Remove temporary prompts, exports, tokens, batches, and model output.

## 2026-07-13 Run

- Questions reviewed: 349
- Additional distractors approved: 1,113
- Final choices: 2,857
- Final pool range: 5-17 choices
- Questions retaining no additional distractor after audit: 28
- Focused coverage repairs approved: 16
- Quality waivers: 3
- Server validator: 349/349 passed
- Production exact comparison: 349/349 passed
- Live pool-aware sample: 38/40 had more than five choices; all included the
  correct answer
- Live legacy sample: 40/40 returned five choices and included the correct
  answer

The quality waivers covered severe hypercalcemia antiresorptive therapy,
residual gastric-cancer risk after *H pylori* eradication, and unspecified oral
antifungal toxicity. In each case, forcing the category would have created a
co-correct or otherwise ambiguous option.