step3/QBANK_PDF_IMPORT_RUNBOOK.md

# Question Bank PDF Import Runbook

This runbook describes how to turn an image-based board-review PDF into
original Step 3-style questions, verify them, insert them through the Questions
API, and confirm the production result. It records the workflow used for the
July 2026 NBME 5 import and strengthens it with the repository's current
reference, originality, image, and duplicate-review requirements.

The source material is used only to identify tested concepts, reasoning paths,
and plausible distractors. Do not copy source wording, vignette structure,
distinctive fact combinations, answer wording, or images.

## July 2026 Result

- 221 image-only PDF pages were rendered and OCRed.
- Page markers were mapped to 176 source questions across five sections.
- Questions were generated in batches of four.
- The first 72 questions passed the initial hosted-model workflow.
- Of the remaining 104, an independent audit approved 95 and rejected 9.
- The 9 rejected questions were regenerated from scratch with the audit issues
  supplied, then all 9 passed a second independent audit.
- The bulk API inserted 104 remaining records, bringing the imported set to
  176 records.
- Direct database verification found 0 missing external IDs and 0 answer-key
  mismatches.

Known gap in that historical run: model-based medical review was completed, but
authoritative references were not embedded in the `nbme5-*` question documents.
Future runs must complete the reference-review gate below before setting
`isPublished` to `true`. The historical records should be treated as awaiting
that separate reference review.

## Prerequisites

- Python 3 with `PyMuPDF`, `Pillow`, and `pytesseract`.
- A local Tesseract OCR installation available on `PATH`.
- PowerShell, `curl`, the .NET SDK, and access to this repository.
- Access to a capable generation model and a separate audit model.
- A `SUPER` JWT for the target Questions API.
- Optional direct MongoDB read access for final production verification.

Keep secrets out of scripts, prompts, logs, and the repository. Configure them
in the current process:

```powershell
$env:QUESTIONS_API_BASE = "https://example.invalid"
$env:QUESTIONS_SUPER_JWT = "<short-lived SUPER token>"
$env:QUESTIONS_MONGODB = "<optional read-only connection string>"
```

Create all intermediate files outside the repository:

```powershell
$RunId = Get-Date -Format "yyyyMMdd-HHmmss"
$WorkDir = Join-Path $env:TEMP "step3-qbank-import-$RunId"
New-Item -ItemType Directory -Path $WorkDir | Out-Null
"ocr", "mapping", "prompts", "generated", "audits", "repairs" |
  ForEach-Object { New-Item -ItemType Directory -Path (Join-Path $WorkDir $_) | Out-Null }
```

Suggested layout:

```text
step3-qbank-import-<run-id>/
  ocr/pages.jsonl
  mapping/items.jsonl
  prompts/
  generated/
  audits/
  repairs/
  validation.json
  references.json
  insert-payload.json
```

## 1. Establish The Import Set

1. Hash and record the source PDF before processing.
2. Record the expected source, section, and question counts if known.
3. Query existing external IDs before generation. Never assume the database is
   empty, and never overwrite an existing ID unless that update is intentional.
4. Choose a deterministic ID prefix, for example
   `nbme5-s3-i007-<concept-slug>`.
5. Keep the original PDF unchanged and outside generated-output directories.

The bulk endpoint upserts by `externalId`. Deterministic IDs make retries
idempotent, but they also make accidental overwrites possible. Compare the
planned IDs with production before insertion.

## 2. Render And OCR The PDF

For an image-only PDF, render each page at approximately 250-300 DPI and run
Tesseract on the rendered bitmap. A minimal Python outline is:

```powershell
py -m pip install pymupdf pillow pytesseract
```

```python
import io, json, sys, fitz, pytesseract
from PIL import Image

source_pdf = sys.argv[1]
pages_jsonl = sys.argv[2]
document = fitz.open(source_pdf)
with open(pages_jsonl, "w", encoding="utf-8") as output:
    for page_number, page in enumerate(document, start=1):
        pixmap = page.get_pixmap(matrix=fitz.Matrix(3, 3), alpha=False)
        image = Image.open(io.BytesIO(pixmap.tobytes("png")))
        text = pytesseract.image_to_string(image, config="--psm 6")
        output.write(json.dumps({
            "page": page_number,
            "text": text,
        }, ensure_ascii=False) + "\n")
```

Save the snippet as `ocr_pdf.py` in the working directory and run:

```powershell
py "$WorkDir\ocr_pdf.py" "C:\path\source.pdf" "$WorkDir\ocr\pages.jsonl"
```

Do not silently discard low-confidence or empty pages. Save page numbers and OCR
text so every generated question remains traceable to its source pages during
review.

## 3. Map Pages To Source Questions

Parse repeated headers such as `Exam Section N: Item M of K`. Group all pages
for the same section/item pair and append continuation pages to the preceding
item when necessary.

Each `items.jsonl` record should contain:

```json
{"section":3,"item":7,"pages":[95],"sourceText":"..."}
```

Before generation, verify:

- every detected section/item pair is unique;
- the item sequence has no unexplained gaps;
- multipage stems and answer choices are complete;
- the total mapped-question count matches a manual review;
- OCR did not convert option letters, decimal points, inequality signs, units,
  or answer-choice text into unusable content;
- image-dependent questions are identified for special handling.

The July 2026 source had 221 pages but only 176 questions. Page count is not a
question-count proxy.

## 4. Generate Original Questions

Generate small batches, typically four source items per request. Small batches
limit loss when a model call fails and make outputs easier to review.

The generation prompt must require:

- a substantially original clinical scenario testing only the same concept;
- changes to age, sex when clinically possible, setting, chronology, numeric
  values, sentence sequence, and answer wording;
- one defensible best answer and a variable pool of plausible choices in an
  array of `{id, text}` objects, not a map keyed by letters;
- a stem containing only the vignette and a separate `leadIn`;
- `choiceExplanations` with exactly one entry for every main choice;
- `wrongAnswerFollowups` for every incorrect main answer;
- three A-C follow-up choices, all of which restate true facts explicitly found
  in the new stem or displayed labs;
- Celsius and Fahrenheit together for every temperature;
- labs only when useful and only when their values also appear in the stem;
- `media: []` unless separately reviewed, licensed media will be added later;
- a `references` array that the reference-review gate populates with complete
  authoritative source records before publication;
- selection percentages for every authored choice initialized to zero;
- established Step categories, 3-6 tags, and a unique deterministic ID.

For image-dependent source items, rewrite the concept as a text-based question
when that can be done without changing the learning objective. Otherwise stop
and perform the image copyright workflow in `README.md`; do not import the
source image.

Run a four-question gate before scaling out. Inspect it manually for
originality, ambiguity, schema shape, medical accuracy, and stem-anchored
follow-ups. Reject the model or revise the prompt if this gate fails.

## 5. Run Deterministic Validation

Combine generated batches only after each parses as JSON. The validator should
fail the run for any of the following:

- missing, duplicate, or incorrectly prefixed external IDs;
- a count different from the mapped source count;
- fewer than five main choices, more than twenty, duplicate choice IDs, or a
  main choice not represented as an `{id, text}` object;
- an invalid `correctChoiceId`;
- missing choice explanations or a correct explanation that does not begin
  with `Correct.`;
- follow-up keys that do not exactly equal the incorrect main-answer IDs;
- follow-up choices not represented by an ordered A-C array;
- follow-up text that cannot be traced to the new stem or labs;
- nonzero initial selection percentages;
- invalid difficulty, category, tag count, publication state, or media shape;
- a Celsius temperature without its Fahrenheit equivalent;
- labs referenced by explanations or follow-ups but absent from the stem;
- suspiciously long contiguous overlap with source wording.

Treat automated overlap and anchoring warnings as review queues, not automatic
approval or rejection. Record the disposition of every warning.

## 6. Perform Independent Content Review

Use a different model or reviewer from the generator. Provide the OCR source
and generated question, and request a machine-readable report:

```json
{"externalId":"...","approved":false,"issues":["concrete issue"]}
```

The reviewer must check:

1. The new item tests the same core reasoning path.
2. It is structurally and textually original.
3. The keyed answer is correct under current practice.
4. No distractor is reasonably co-correct.
5. All explanations are accurate.
6. Every follow-up choice is an explicit fact from the generated stem or labs.
7. The vignette is internally consistent and sufficient to answer.

Regenerate rejected items from scratch with the audit issues included. Do not
patch only the reported sentence when the underlying scenario is ambiguous.
Audit every repaired item again. Merge a repair only after it receives an
independent approval with an empty issue list.

## 7. Complete Required Reference And Duplicate Reviews

Model agreement is not authoritative medical verification. Before publication:

1. Verify each keyed answer and material explanatory claim against current,
   direct, authoritative sources such as professional guidelines, government
   agencies, official labeling, or primary research.
2. Add every verified source to the question document's `references` array,
   including title, publisher, direct HTTPS URL, supported claim, ISO access
   date, and reviewer. Do not maintain a separate repository reference ledger.
3. Search the entire question bank for exact and near-duplicate learning tasks,
   not just duplicate text or IDs.
4. Record duplicate-review dispositions and substantially differentiate or
   remove near-duplicates.
5. Complete the image licensing review for every non-empty `media` field.

Keep new questions unpublished until these gates are complete. If generation
produces `isPublished: true`, change it to `false` before insertion and publish
only after the reviews are recorded.

## 8. Build And Submit The Payload

The API expects this wrapper:

```json
{"questions":[{...},{...}]}
```

Submit the validated file exactly as UTF-8 bytes:

```powershell
curl.exe --fail-with-body `
  -X POST "$env:QUESTIONS_API_BASE/api/questions/bulk" `
  -H "Authorization: Bearer $env:QUESTIONS_SUPER_JWT" `
  -H "Content-Type: application/json; charset=utf-8" `
  --data-binary "@${WorkDir}\insert-payload.json"
```

Expect `{"upserted":N}` with HTTP 200. PowerShell 5.1 can transmit a string body
with an unexpected encoding; prefer `curl --data-binary` or pass explicit UTF-8
bytes to `Invoke-RestMethod`.

Do not insert partial batches merely because generation is expensive. Complete
the validation, review, repair, reference, and duplicate gates first.

## 9. Verify Production

Verification is complete only when all of these pass:

- the API reports the expected upsert count;
- the database contains every expected external ID exactly once;
- every stored `correctChoiceId` matches the final payload;
- every stored `references` array exactly matches the verified payload;
- a targeted deployed-API search returns at least one newly inserted question;
- the returned question has the expected choices, answer key, and publication
  state;
- repository state is unchanged unless documentation or tooling was
  intentionally committed.

Do not use an unfiltered search to prove completeness. The API limits search
results and randomizes the returned order. Compare IDs directly through a
read-only database query or a dedicated administrative verification tool, then
use a highly specific tag combination for the deployed-API read check.

The final verification report should include at least:

```text
EXPECTED=<count>
STORED=<count>
MISSING=<count>
DUPLICATES=<count>
ANSWER_MISMATCHES=<count>
API_READ_RESULTS=<count>
```

## 10. Retain Reviews, Then Clean Up

Retain authoritative references in each question document. Retain the
duplicate-review record, validation summary, audit decisions, and final
ID/answer-key manifest according to project policy. These are review records,
not disposable generation scratch files.

After production verification:

1. Delete rendered page images, raw model logs, rejected generations, temporary
   prompts, local test projects, and disposable payload copies.
2. Preserve or securely dispose of the original PDF according to its license
   and organizational policy.
3. Remove large local models installed only for the import.
4. Confirm the repository is clean with `git status --short --branch`.
5. Commit and push only maintained documentation, tooling, and review records.

Use explicit, workspace-contained paths for cleanup. Resolve each path before a
recursive deletion and never delete the repository or source PDF as part of a
generated-artifact cleanup.

## Acceptance Checklist

- [ ] Source PDF hash and import scope recorded.
- [ ] OCR page count and mapped-question count reconciled.
- [ ] Every source item mapped exactly once.
- [ ] Four-question generation gate passed.
- [ ] Full schema and ID validation passed with zero errors.
- [ ] All anchoring and source-overlap warnings reviewed.
- [ ] Independent content audit approved every question.
- [ ] Every rejected question regenerated and re-audited.
- [ ] Authoritative references embedded in every question document.
- [ ] Full-bank duplicate review completed.
- [ ] Image rights cleared for every media-bearing question.
- [ ] Final payload contains only intended new or intentionally updated IDs.
- [ ] Bulk API returned the expected upsert count.
- [ ] Database has zero missing IDs and answer-key mismatches.
- [ ] Deployed API read check passed.
- [ ] Required review records retained and scratch files removed.