# UAT and production operations

## Environment map

| Environment | Branch | Web | API | Namespace | Mongo database | NodePorts |
| --- | --- | --- | --- | --- | --- | --- |
| UAT | `main` | `https://ustep3.ranaze.com` | `https://us3q.ranaze.com` | `step3-uat` | `ranaze_step3_uat` | API `30088`, web `30089` |
| PRD | `production` | `https://step3.ranaze.com` | `https://s3q.ranaze.com` | `step3-prd` | `ranaze_step3_prd` | API `30090`, web `30093` |

`doctor.ranaze.com` should issue an HTTP redirect to `https://ustep3.ranaze.com`. Traefik and Cloudflare configuration are external to this repository.

`main` is the UAT integration branch. Protect `production`, disable direct pushes, and promote releases through a reviewed pull request from `main`. Both deployment workflows build and deploy the exact branch commit with immutable SHA image tags.

## Required Gitea secrets

The existing `KUBECONFIG` secret is shared. Configure these environment-specific secrets before enabling the new workflows:

- `UAT_MONGODB_CONNECTION_STRING` and `PRD_MONGODB_CONNECTION_STRING`
- `UAT_AUTH_JWT_KEY` and `PRD_AUTH_JWT_KEY` (different values)
- `UAT_INTERNAL_PURGE_SECRET` and `PRD_INTERNAL_PURGE_SECRET`
- `UAT_STRIPE_SECRET_KEY` and `UAT_STRIPE_WEBHOOK_SECRET` (Stripe sandbox)
- `PRD_STRIPE_SECRET_KEY` and `PRD_STRIPE_WEBHOOK_SECRET` (Stripe live mode)

Use independent Stripe webhook endpoints for `https://us3q.ranaze.com/api/donations/stripe-webhook` and `https://s3q.ranaze.com/api/donations/stripe-webhook`. No Stripe publishable key is required for hosted Checkout.

## Initial UAT database migration

Schedule a short maintenance window so writes cannot occur between the dump and cutover.

1. Route the existing app to a maintenance response or scale the legacy API to zero.
2. Back up the current database:

   ```bash
   mongodump --uri="$UAT_MONGODB_CONNECTION_STRING" --db=promiscuity --archive=promiscuity.archive --gzip
   ```

3. Restore it under the explicit UAT name:

   ```bash
   mongorestore --uri="$UAT_MONGODB_CONNECTION_STRING" --archive=promiscuity.archive --gzip --nsFrom='promiscuity.*' --nsTo='ranaze_step3_uat.*'
   ```

4. Compare collection/document counts between `promiscuity` and `ranaze_step3_uat` and retain the archive until acceptance testing is complete.
5. The legacy services currently reserve NodePorts `30088` and `30089`. Immediately before the first UAT deployment, release them:

   ```bash
   kubectl delete service step3-questions -n step3-questions
   kubectl delete service step3-web -n step3-web
   ```

6. Deploy `main`, route `ustep3` and `us3q` to the UAT NodePorts, verify login and study activity, then redirect `doctor` to UAT.
7. After acceptance, remove the superseded `step3-questions` and `step3-web` namespaces. Do not remove them before retaining any logs needed for migration review.

The hostname change creates a new browser origin, so testers should expect to sign in again. Their accounts and activity remain in UAT.

## Initial PRD launch

1. Create and protect the `production` branch, then merge the tested `main` revision.
2. Route `step3.ranaze.com` and `s3q.ranaze.com` to PRD NodePorts.
3. Run **Promote ready UAT questions to PRD** in `dry-run` mode and inspect the insert, update, unchanged, extra, media, and hash values.
4. Run it again in `apply` mode with confirmation `PROMOTE_TO_PRD`.
5. Verify `ranaze_step3_prd` contains the published, unflagged UAT questions and has no user/activity documents.
6. Register the initial PRD account normally, then promote it manually:

   ```javascript
   use ranaze_step3_prd
   db.Users.updateOne({Username: "SUPER"}, {$set: {Role: "SUPER"}})
   ```

7. Configure and test the live Stripe webhook and Steam callback before announcing PRD.

## Question promotion behavior

The manual workflow runs the deployed PRD API image as a short-lived pod. It copies only questions that are published and not flagged for review, plus their referenced media. Any promoted question containing media is flagged again in PRD and remains hidden from students until a SUPER user explicitly approves it in production. UAT media approval is not treated as production approval. It never copies CCS content, users, attempts, ratings, statistics, simulations, reports, feedback, notes, or donations.

Promotion is idempotent and does not delete or implicitly unpublish PRD content. Its dry run reports inserts, updates, unchanged questions, and production extras. Apply mode verifies the promoted target records and writes a scoped `ContentPromotions` audit document. Corrective unpublishing remains an explicit SUPER/API or database operation.

## Verification and rollback

## Step 3 mastery system backfill

The mastery dashboard uses a canonical 14-system classification stored on each
question. After deploying the mastery-capable API, inspect and apply the
idempotent backfill in UAT:

```bash
kubectl exec deployment/step3-api -n step3-uat -- \
  dotnet QuestionsApi.dll backfill-step3-systems
kubectl exec deployment/step3-api -n step3-uat -- \
  dotnet QuestionsApi.dll backfill-step3-systems --apply --confirm BACKFILL_STEP3_SYSTEMS
```

Review the reported unmapped count before acceptance testing. Repeat the dry run
and confirmed apply in `step3-prd` after the tested revision is promoted to the
production branch. The command is safe to rerun and does not modify attempts or
user data.

## Rz Coin Store attempt-cycle migration

The QBank reset preserves every attempt and increments a user's eligibility cycle. After deploying
the store-capable API to an environment, inspect and then apply the one-time index migration before
using QBank reset tokens:

```bash
kubectl exec deployment/step3-api -n step3-uat -- \
  dotnet QuestionsApi.dll migrate-question-attempt-cycles
kubectl exec deployment/step3-api -n step3-uat -- \
  dotnet QuestionsApi.dll migrate-question-attempt-cycles --apply --confirm MIGRATE_ATTEMPT_CYCLES
```

Repeat with namespace `step3-prd` after the tested revision is promoted to `production`. The dry run
reports missing cycle fields and both index states. Apply backfills existing attempts to cycle zero,
creates the three-field unique index, and only then removes the legacy two-field unique index.

Check both internal and public readiness after every rollout:

```bash
kubectl get pods -n step3-uat
kubectl get pods -n step3-prd
kubectl get --raw /api/v1/namespaces/step3-uat/services/step3-api:http/proxy/readyz
kubectl get --raw /api/v1/namespaces/step3-prd/services/step3-api:http/proxy/readyz
curl -fsS https://us3q.ranaze.com/api/runtime
curl -fsS https://s3q.ranaze.com/api/runtime
```

Rollback application code by setting each deployment back to a previously imported SHA-tagged image. Database promotion is non-destructive; restore the pre-cutover archive if the initial UAT migration itself must be reversed.