127 lines
6.7 KiB
Markdown
127 lines
6.7 KiB
Markdown
# UAT and production operations
|
|
|
|
## Environment map
|
|
|
|
| Environment | Branch | Web | API | Namespace | Mongo database | NodePorts |
|
|
| --- | --- | --- | --- | --- | --- | --- |
|
|
| UAT | `main` | `https://ustep3.ranaze.com` | `https://us3q.ranaze.com` | `step3-uat` | `ranaze_step3_uat` | API `30088`, web `30089` |
|
|
| PRD | `production` | `https://step3.ranaze.com` | `https://s3q.ranaze.com` | `step3-prd` | `ranaze_step3_prd` | API `30090`, web `30093` |
|
|
|
|
`doctor.ranaze.com` should issue an HTTP redirect to `https://ustep3.ranaze.com`. Traefik and Cloudflare configuration are external to this repository.
|
|
|
|
`main` is the UAT integration branch. Protect `production`, disable direct pushes, and promote releases through a reviewed pull request from `main`. Both deployment workflows build and deploy the exact branch commit with immutable SHA image tags.
|
|
|
|
## Required Gitea secrets
|
|
|
|
The existing `KUBECONFIG` secret is shared. Configure these environment-specific secrets before enabling the new workflows:
|
|
|
|
- `UAT_MONGODB_CONNECTION_STRING` and `PRD_MONGODB_CONNECTION_STRING`
|
|
- `UAT_AUTH_JWT_KEY` and `PRD_AUTH_JWT_KEY` (different values)
|
|
- `UAT_INTERNAL_PURGE_SECRET` and `PRD_INTERNAL_PURGE_SECRET`
|
|
- `UAT_STRIPE_SECRET_KEY` and `UAT_STRIPE_WEBHOOK_SECRET` (Stripe sandbox)
|
|
- `PRD_STRIPE_SECRET_KEY` and `PRD_STRIPE_WEBHOOK_SECRET` (Stripe live mode)
|
|
|
|
Use independent Stripe webhook endpoints for `https://us3q.ranaze.com/api/donations/stripe-webhook` and `https://s3q.ranaze.com/api/donations/stripe-webhook`. No Stripe publishable key is required for hosted Checkout.
|
|
|
|
## Initial UAT database migration
|
|
|
|
Schedule a short maintenance window so writes cannot occur between the dump and cutover.
|
|
|
|
1. Route the existing app to a maintenance response or scale the legacy API to zero.
|
|
2. Back up the current database:
|
|
|
|
```bash
|
|
mongodump --uri="$UAT_MONGODB_CONNECTION_STRING" --db=promiscuity --archive=promiscuity.archive --gzip
|
|
```
|
|
|
|
3. Restore it under the explicit UAT name:
|
|
|
|
```bash
|
|
mongorestore --uri="$UAT_MONGODB_CONNECTION_STRING" --archive=promiscuity.archive --gzip --nsFrom='promiscuity.*' --nsTo='ranaze_step3_uat.*'
|
|
```
|
|
|
|
4. Compare collection/document counts between `promiscuity` and `ranaze_step3_uat` and retain the archive until acceptance testing is complete.
|
|
5. The legacy services currently reserve NodePorts `30088` and `30089`. Immediately before the first UAT deployment, release them:
|
|
|
|
```bash
|
|
kubectl delete service step3-questions -n step3-questions
|
|
kubectl delete service step3-web -n step3-web
|
|
```
|
|
|
|
6. Deploy `main`, route `ustep3` and `us3q` to the UAT NodePorts, verify login and study activity, then redirect `doctor` to UAT.
|
|
7. After acceptance, remove the superseded `step3-questions` and `step3-web` namespaces. Do not remove them before retaining any logs needed for migration review.
|
|
|
|
The hostname change creates a new browser origin, so testers should expect to sign in again. Their accounts and activity remain in UAT.
|
|
|
|
## Initial PRD launch
|
|
|
|
1. Create and protect the `production` branch, then merge the tested `main` revision.
|
|
2. Route `step3.ranaze.com` and `s3q.ranaze.com` to PRD NodePorts.
|
|
3. Run **Promote ready UAT questions to PRD** in `dry-run` mode and inspect the insert, update, unchanged, extra, media, and hash values.
|
|
4. Run it again in `apply` mode with confirmation `PROMOTE_TO_PRD`.
|
|
5. Verify `ranaze_step3_prd` contains the published, unflagged UAT questions and has no user/activity documents.
|
|
6. Register the initial PRD account normally, then promote it manually:
|
|
|
|
```javascript
|
|
use ranaze_step3_prd
|
|
db.Users.updateOne({Username: "SUPER"}, {$set: {Role: "SUPER"}})
|
|
```
|
|
|
|
7. Configure and test the live Stripe webhook and Steam callback before announcing PRD.
|
|
|
|
## Question promotion behavior
|
|
|
|
The manual workflow runs the deployed PRD API image as a short-lived pod. It copies only questions that are published and not flagged for review, plus their referenced media. Any promoted question containing media is flagged again in PRD and remains hidden from students until a SUPER user explicitly approves it in production. UAT media approval is not treated as production approval. It never copies CCS content, users, attempts, ratings, statistics, simulations, reports, feedback, notes, or donations.
|
|
|
|
Promotion is idempotent and does not delete or implicitly unpublish PRD content. Its dry run reports inserts, updates, unchanged questions, and production extras. Apply mode verifies the promoted target records and writes a scoped `ContentPromotions` audit document. Corrective unpublishing remains an explicit SUPER/API or database operation.
|
|
|
|
## Verification and rollback
|
|
|
|
## Step 3 mastery system backfill
|
|
|
|
The mastery dashboard uses a canonical 14-system classification stored on each
|
|
question. After deploying the mastery-capable API, inspect and apply the
|
|
idempotent backfill in UAT:
|
|
|
|
```bash
|
|
kubectl exec deployment/step3-api -n step3-uat -- \
|
|
dotnet QuestionsApi.dll backfill-step3-systems
|
|
kubectl exec deployment/step3-api -n step3-uat -- \
|
|
dotnet QuestionsApi.dll backfill-step3-systems --apply --confirm BACKFILL_STEP3_SYSTEMS
|
|
```
|
|
|
|
Review the reported unmapped count before acceptance testing. Repeat the dry run
|
|
and confirmed apply in `step3-prd` after the tested revision is promoted to the
|
|
production branch. The command is safe to rerun and does not modify attempts or
|
|
user data.
|
|
|
|
## Rz Coin Store attempt-cycle migration
|
|
|
|
The QBank reset preserves every attempt and increments a user's eligibility cycle. After deploying
|
|
the store-capable API to an environment, inspect and then apply the one-time index migration before
|
|
using QBank reset tokens:
|
|
|
|
```bash
|
|
kubectl exec deployment/step3-api -n step3-uat -- \
|
|
dotnet QuestionsApi.dll migrate-question-attempt-cycles
|
|
kubectl exec deployment/step3-api -n step3-uat -- \
|
|
dotnet QuestionsApi.dll migrate-question-attempt-cycles --apply --confirm MIGRATE_ATTEMPT_CYCLES
|
|
```
|
|
|
|
Repeat with namespace `step3-prd` after the tested revision is promoted to `production`. The dry run
|
|
reports missing cycle fields and both index states. Apply backfills existing attempts to cycle zero,
|
|
creates the three-field unique index, and only then removes the legacy two-field unique index.
|
|
|
|
Check both internal and public readiness after every rollout:
|
|
|
|
```bash
|
|
kubectl get pods -n step3-uat
|
|
kubectl get pods -n step3-prd
|
|
kubectl get --raw /api/v1/namespaces/step3-uat/services/step3-api:http/proxy/readyz
|
|
kubectl get --raw /api/v1/namespaces/step3-prd/services/step3-api:http/proxy/readyz
|
|
curl -fsS https://us3q.ranaze.com/api/runtime
|
|
curl -fsS https://s3q.ranaze.com/api/runtime
|
|
```
|
|
|
|
Rollback application code by setting each deployment back to a previously imported SHA-tagged image. Database promotion is non-destructive; restore the pre-cutover archive if the initial UAT migration itself must be reversed.
|