step3/QBANK_PDF_IMPORT_RUNBOOK.md

Question Bank PDF Import Runbook

This runbook describes how to turn an image-based board-review PDF into original Step 3-style questions, verify them, insert them through the Questions API, and confirm the production result. It records the workflow used for the July 2026 NBME 5 import and strengthens it with the repository's current reference, originality, image, and duplicate-review requirements.

The source material is used only to identify tested concepts, reasoning paths, and plausible distractors. Do not copy source wording, vignette structure, distinctive fact combinations, answer wording, or images.

July 2026 Result

• 221 image-only PDF pages were rendered and OCRed.
• Page markers were mapped to 176 source questions across five sections.
• Questions were generated in batches of four.
• The first 72 questions passed the initial hosted-model workflow.
• Of the remaining 104, an independent audit approved 95 and rejected 9.
• The 9 rejected questions were regenerated from scratch with the audit issues supplied, then all 9 passed a second independent audit.
• The bulk API inserted 104 remaining records, bringing the imported set to 176 records.
• Direct database verification found 0 missing external IDs and 0 answer-key mismatches.

Known gap in that historical run: model-based medical review was completed, but authoritative references were not embedded in the nbme5-* question documents. Future runs must complete the reference-review gate below before setting isPublished to true. The historical records should be treated as awaiting that separate reference review.

Prerequisites

• Python 3 with PyMuPDF, Pillow, and pytesseract.
• A local Tesseract OCR installation available on PATH.
• PowerShell, curl, the .NET SDK, and access to this repository.
• Access to a capable generation model and a separate audit model.
• A SUPER JWT for the target Questions API.
• Optional direct MongoDB read access for final production verification.

Keep secrets out of scripts, prompts, logs, and the repository. Configure them in the current process:

$env:QUESTIONS_API_BASE = "https://example.invalid"
$env:QUESTIONS_SUPER_JWT = "<short-lived SUPER token>"
$env:QUESTIONS_MONGODB = "<optional read-only connection string>"

Create all intermediate files outside the repository:

$RunId = Get-Date -Format "yyyyMMdd-HHmmss"
$WorkDir = Join-Path $env:TEMP "step3-qbank-import-$RunId"
New-Item -ItemType Directory -Path $WorkDir | Out-Null
"ocr", "mapping", "prompts", "generated", "audits", "repairs" |
  ForEach-Object { New-Item -ItemType Directory -Path (Join-Path $WorkDir $_) | Out-Null }

Suggested layout:

step3-qbank-import-<run-id>/
  ocr/pages.jsonl
  mapping/items.jsonl
  prompts/
  generated/
  audits/
  repairs/
  validation.json
  references.json
  insert-payload.json

1. Establish The Import Set

1. Hash and record the source PDF before processing.
2. Record the expected source, section, and question counts if known.
3. Query existing external IDs before generation. Never assume the database is empty, and never overwrite an existing ID unless that update is intentional.
4. Choose a deterministic ID prefix, for example nbme5-s3-i007-<concept-slug>.
5. Keep the original PDF unchanged and outside generated-output directories.

The bulk endpoint upserts by externalId. Deterministic IDs make retries idempotent, but they also make accidental overwrites possible. Compare the planned IDs with production before insertion.

2. Render And OCR The PDF

For an image-only PDF, render each page at approximately 250-300 DPI and run Tesseract on the rendered bitmap. A minimal Python outline is:

py -m pip install pymupdf pillow pytesseract

import io, json, sys, fitz, pytesseract
from PIL import Image

source_pdf = sys.argv[1]
pages_jsonl = sys.argv[2]
document = fitz.open(source_pdf)
with open(pages_jsonl, "w", encoding="utf-8") as output:
    for page_number, page in enumerate(document, start=1):
        pixmap = page.get_pixmap(matrix=fitz.Matrix(3, 3), alpha=False)
        image = Image.open(io.BytesIO(pixmap.tobytes("png")))
        text = pytesseract.image_to_string(image, config="--psm 6")
        output.write(json.dumps({
            "page": page_number,
            "text": text,
        }, ensure_ascii=False) + "\n")

Save the snippet as ocr_pdf.py in the working directory and run:

py "$WorkDir\ocr_pdf.py" "C:\path\source.pdf" "$WorkDir\ocr\pages.jsonl"

Do not silently discard low-confidence or empty pages. Save page numbers and OCR text so every generated question remains traceable to its source pages during review.

3. Map Pages To Source Questions

Parse repeated headers such as Exam Section N: Item M of K. Group all pages for the same section/item pair and append continuation pages to the preceding item when necessary.

Each items.jsonl record should contain:

{"section":3,"item":7,"pages":[95],"sourceText":"..."}

Before generation, verify:

• every detected section/item pair is unique;
• the item sequence has no unexplained gaps;
• multipage stems and answer choices are complete;
• the total mapped-question count matches a manual review;
• OCR did not convert option letters, decimal points, inequality signs, units, or answer-choice text into unusable content;
• image-dependent questions are identified for special handling.

The July 2026 source had 221 pages but only 176 questions. Page count is not a question-count proxy.

4. Generate Original Questions

Generate small batches, typically four source items per request. Small batches limit loss when a model call fails and make outputs easier to review.

The generation prompt must require:

• a substantially original clinical scenario testing only the same concept;
• changes to age, sex when clinically possible, setting, chronology, numeric values, sentence sequence, and answer wording;
• one defensible best answer and a variable pool of plausible choices in an array of {id, text} objects, not a map keyed by letters;
• a stem containing only the vignette and a separate leadIn;
• choiceExplanations with exactly one entry for every main choice;
• wrongAnswerFollowups for every incorrect main answer;
• three A-C follow-up choices, all of which restate true facts explicitly found in the new stem or displayed labs;
• Celsius and Fahrenheit together for every temperature;
• labs only when useful and only when their values also appear in the stem;
• media: [] unless separately reviewed, licensed media will be added later;
• a references array that the reference-review gate populates with complete authoritative source records before publication;
• selection percentages for every authored choice initialized to zero;
• established Step categories, 3-6 tags, and a unique deterministic ID.

For image-dependent source items, rewrite the concept as a text-based question when that can be done without changing the learning objective. Otherwise stop and perform the image copyright workflow in README.md; do not import the source image.

Run a four-question gate before scaling out. Inspect it manually for originality, ambiguity, schema shape, medical accuracy, and stem-anchored follow-ups. Reject the model or revise the prompt if this gate fails.

5. Run Deterministic Validation

Combine generated batches only after each parses as JSON. The validator should fail the run for any of the following:

• missing, duplicate, or incorrectly prefixed external IDs;
• a count different from the mapped source count;
• fewer than five main choices, more than twenty, duplicate choice IDs, or a main choice not represented as an {id, text} object;
• an invalid correctChoiceId;
• missing choice explanations or a correct explanation that does not begin with Correct.;
• follow-up keys that do not exactly equal the incorrect main-answer IDs;
• follow-up choices not represented by an ordered A-C array;
• follow-up text that cannot be traced to the new stem or labs;
• nonzero initial selection percentages;
• invalid difficulty, category, tag count, publication state, or media shape;
• a Celsius temperature without its Fahrenheit equivalent;
• labs referenced by explanations or follow-ups but absent from the stem;
• suspiciously long contiguous overlap with source wording.

Treat automated overlap and anchoring warnings as review queues, not automatic approval or rejection. Record the disposition of every warning.

6. Perform Independent Content Review

Use a different model or reviewer from the generator. Provide the OCR source and generated question, and request a machine-readable report:

{"externalId":"...","approved":false,"issues":["concrete issue"]}

The reviewer must check:

1. The new item tests the same core reasoning path.
2. It is structurally and textually original.
3. The keyed answer is correct under current practice.
4. No distractor is reasonably co-correct.
5. All explanations are accurate.
6. Every follow-up choice is an explicit fact from the generated stem or labs.
7. The vignette is internally consistent and sufficient to answer.

Regenerate rejected items from scratch with the audit issues included. Do not patch only the reported sentence when the underlying scenario is ambiguous. Audit every repaired item again. Merge a repair only after it receives an independent approval with an empty issue list.

7. Complete Required Reference And Duplicate Reviews

Model agreement is not authoritative medical verification. Before publication:

1. Verify each keyed answer and material explanatory claim against current, direct, authoritative sources such as professional guidelines, government agencies, official labeling, or primary research.
2. Add every verified source to the question document's references array, including title, publisher, direct HTTPS URL, supported claim, ISO access date, and reviewer. Do not maintain a separate repository reference ledger.
3. Search the entire question bank for exact and near-duplicate learning tasks, not just duplicate text or IDs.
4. Record duplicate-review dispositions and substantially differentiate or remove near-duplicates.
5. Complete the image licensing review for every non-empty media field.

Keep new questions unpublished until these gates are complete. If generation produces isPublished: true, change it to false before insertion and publish only after the reviews are recorded.

8. Build And Submit The Payload

The API expects this wrapper:

{"questions":[{...},{...}]}

Submit the validated file exactly as UTF-8 bytes:

curl.exe --fail-with-body `
  -X POST "$env:QUESTIONS_API_BASE/api/questions/bulk" `
  -H "Authorization: Bearer $env:QUESTIONS_SUPER_JWT" `
  -H "Content-Type: application/json; charset=utf-8" `
  --data-binary "@${WorkDir}\insert-payload.json"

Expect {"upserted":N} with HTTP 200. PowerShell 5.1 can transmit a string body with an unexpected encoding; prefer curl --data-binary or pass explicit UTF-8 bytes to Invoke-RestMethod.

Do not insert partial batches merely because generation is expensive. Complete the validation, review, repair, reference, and duplicate gates first.

9. Verify Production

Verification is complete only when all of these pass:

• the API reports the expected upsert count;
• the database contains every expected external ID exactly once;
• every stored correctChoiceId matches the final payload;
• every stored references array exactly matches the verified payload;
• a targeted deployed-API search returns at least one newly inserted question;
• the returned question has the expected choices, answer key, and publication state;
• repository state is unchanged unless documentation or tooling was intentionally committed.

Do not use an unfiltered search to prove completeness. The API limits search results and randomizes the returned order. Compare IDs directly through a read-only database query or a dedicated administrative verification tool, then use a highly specific tag combination for the deployed-API read check.

The final verification report should include at least:

EXPECTED=<count>
STORED=<count>
MISSING=<count>
DUPLICATES=<count>
ANSWER_MISMATCHES=<count>
API_READ_RESULTS=<count>

10. Retain Reviews, Then Clean Up

Retain authoritative references in each question document. Retain the duplicate-review record, validation summary, audit decisions, and final ID/answer-key manifest according to project policy. These are review records, not disposable generation scratch files.

After production verification:

1. Delete rendered page images, raw model logs, rejected generations, temporary prompts, local test projects, and disposable payload copies.
2. Preserve or securely dispose of the original PDF according to its license and organizational policy.
3. Remove large local models installed only for the import.
4. Confirm the repository is clean with git status --short --branch.
5. Commit and push only maintained documentation, tooling, and review records.

Use explicit, workspace-contained paths for cleanup. Resolve each path before a recursive deletion and never delete the repository or source PDF as part of a generated-artifact cleanup.

Acceptance Checklist

• Source PDF hash and import scope recorded.
• OCR page count and mapped-question count reconciled.
• Every source item mapped exactly once.
• Four-question generation gate passed.
• Full schema and ID validation passed with zero errors.
• All anchoring and source-overlap warnings reviewed.
• Independent content audit approved every question.
• Every rejected question regenerated and re-audited.
• Authoritative references embedded in every question document.
• Full-bank duplicate review completed.
• Image rights cleared for every media-bearing question.
• Final payload contains only intended new or intentionally updated IDs.
• Bulk API returned the expected upsert count.
• Database has zero missing IDs and answer-key mismatches.
• Deployed API read check passed.
• Required review records retained and scratch files removed.