Make embedded references canonical

This commit is contained in:
hz
2026-07-13 22:32:31 -05:00
parent 292f8ff9b4
commit 37218d9f45
4 changed files with 21 additions and 1366 deletions
+2
View File
@@ -52,6 +52,8 @@ existing IDs. For each question, verify:
for the correct choice; for the correct choice;
- every follow-up has exactly A-C choices and a valid keyed answer; - every follow-up has exactly A-C choices and a valid keyed answer;
- new `choiceSelectionPercentages` entries start at zero; - new `choiceSelectionPercentages` entries start at zero;
- authoritative sources supporting approved additions are merged into the
question's `references` array without duplicating existing URLs;
- no unresolved coverage issue remains unless documented as a waiver. - no unresolved coverage issue remains unless documented as a waiver.
Deserialize the payload as `BulkUpsertQuestionsRequest` and run the actual Deserialize the payload as `BulkUpsertQuestionsRequest` and run the actual
+15 -10
View File
@@ -25,10 +25,10 @@ distinctive fact combinations, answer wording, or images.
mismatches. mismatches.
Known gap in that historical run: model-based medical review was completed, but Known gap in that historical run: model-based medical review was completed, but
authoritative references were not added to `QUESTION_REFERENCE_REVIEWS.md` for authoritative references were not embedded in the `nbme5-*` question documents.
the `nbme5-*` records. Future runs must complete the reference-review gate below Future runs must complete the reference-review gate below before setting
before setting `isPublished` to `true`. The historical records should be treated `isPublished` to `true`. The historical records should be treated as awaiting
as awaiting that separate reference review. that separate reference review.
## Prerequisites ## Prerequisites
@@ -169,6 +169,8 @@ The generation prompt must require:
- Celsius and Fahrenheit together for every temperature; - Celsius and Fahrenheit together for every temperature;
- labs only when useful and only when their values also appear in the stem; - labs only when useful and only when their values also appear in the stem;
- `media: []` unless separately reviewed, licensed media will be added later; - `media: []` unless separately reviewed, licensed media will be added later;
- a `references` array that the reference-review gate populates with complete
authoritative source records before publication;
- selection percentages for every authored choice initialized to zero; - selection percentages for every authored choice initialized to zero;
- established Step categories, 3-6 tags, and a unique deterministic ID. - established Step categories, 3-6 tags, and a unique deterministic ID.
@@ -236,8 +238,9 @@ Model agreement is not authoritative medical verification. Before publication:
1. Verify each keyed answer and material explanatory claim against current, 1. Verify each keyed answer and material explanatory claim against current,
direct, authoritative sources such as professional guidelines, government direct, authoritative sources such as professional guidelines, government
agencies, official labeling, or primary research. agencies, official labeling, or primary research.
2. Add an entry for every question to `QUESTION_REFERENCE_REVIEWS.md`, including 2. Add every verified source to the question document's `references` array,
title, publisher, direct URL, supported claim, access date, and reviewer. including title, publisher, direct HTTPS URL, supported claim, ISO access
date, and reviewer. Do not maintain a separate repository reference ledger.
3. Search the entire question bank for exact and near-duplicate learning tasks, 3. Search the entire question bank for exact and near-duplicate learning tasks,
not just duplicate text or IDs. not just duplicate text or IDs.
4. Record duplicate-review dispositions and substantially differentiate or 4. Record duplicate-review dispositions and substantially differentiate or
@@ -280,6 +283,7 @@ Verification is complete only when all of these pass:
- the API reports the expected upsert count; - the API reports the expected upsert count;
- the database contains every expected external ID exactly once; - the database contains every expected external ID exactly once;
- every stored `correctChoiceId` matches the final payload; - every stored `correctChoiceId` matches the final payload;
- every stored `references` array exactly matches the verified payload;
- a targeted deployed-API search returns at least one newly inserted question; - a targeted deployed-API search returns at least one newly inserted question;
- the returned question has the expected choices, answer key, and publication - the returned question has the expected choices, answer key, and publication
state; state;
@@ -304,9 +308,10 @@ API_READ_RESULTS=<count>
## 10. Retain Reviews, Then Clean Up ## 10. Retain Reviews, Then Clean Up
Retain the reference ledger, duplicate-review record, validation summary, audit Retain authoritative references in each question document. Retain the
decisions, and final ID/answer-key manifest according to project policy. These duplicate-review record, validation summary, audit decisions, and final
are review records, not disposable generation scratch files. ID/answer-key manifest according to project policy. These are review records,
not disposable generation scratch files.
After production verification: After production verification:
@@ -332,7 +337,7 @@ generated-artifact cleanup.
- [ ] All anchoring and source-overlap warnings reviewed. - [ ] All anchoring and source-overlap warnings reviewed.
- [ ] Independent content audit approved every question. - [ ] Independent content audit approved every question.
- [ ] Every rejected question regenerated and re-audited. - [ ] Every rejected question regenerated and re-audited.
- [ ] Authoritative references recorded for every question. - [ ] Authoritative references embedded in every question document.
- [ ] Full-bank duplicate review completed. - [ ] Full-bank duplicate review completed.
- [ ] Image rights cleared for every media-bearing question. - [ ] Image rights cleared for every media-bearing question.
- [ ] Final payload contains only intended new or intentionally updated IDs. - [ ] Final payload contains only intended new or intentionally updated IDs.
File diff suppressed because it is too large Load Diff
+4 -3
View File
@@ -104,9 +104,10 @@ For each question, document:
References must support the keyed answer and any management recommendations, References must support the keyed answer and any management recommendations,
diagnostic thresholds, formulas, or numerical claims. A citation does not grant diagnostic thresholds, formulas, or numerical claims. A citation does not grant
permission to copy wording or media; the separate text-originality and image- permission to copy wording or media; the separate text-originality and image-
copyright requirements still apply. If the question schema does not expose a copyright requirements still apply. Store each verified source in the question
reference field, retain this information in the associated authoring or review document's `references` array; published question writes require at least one
record until it can be stored with the question. complete reference. Legacy questions with an empty array remain readable while
awaiting reference review.
## Variable Distractor Pools ## Variable Distractor Pools