Backup and restore

The database and the documents bucket are backed up continuously; the design states a recovery point of 5 minutes and a recovery time of 4 hours, and a restore test on a scratch cluster repeats quarterly. The test matters twice: it proves the backups restore, and it recomputes the audit hashes and signature validity on the restored copy, which is evidence the light validation package records. This page is designed until the R2 gate; the numbers are design targets from the plan, not measurements.

Nothing here is a compliance claim. The runbook describes the controls (continuous backup, snapshot copy, object versioning, hash recomputation) so a reviewer can inspect them.

What is backed up

Store	Mechanism	Retention
Database (Aurora PostgreSQL 16, Serverless v2)	automated continuous backups plus a daily snapshot copy to a second bucket	35 days of automated backups; daily copies per the environment’s policy
Documents bucket `trovensa-<env>-documents`	object versioning with server-side encryption; prefix `<org_id>/<program_id>/`	versions kept per the bucket lifecycle
Secrets	the secret manager’s own versioning; two run-token kids during rotation	per the 90-day rotation

Recovery objectives stated by the design: RPO 5 minutes (continuous backup), RTO 4 hours (documented restore).

Restore test (quarterly)

Restore the latest automated backup to a scratch cluster in the same region; restore the documents bucket to a scratch bucket at the matching point in time.
Point a scratch API and worker (application role DSN) at the copy; do not expose them.
Recompute the audit trail: export every organisation’s rows and run the verifier (verify_row_hash from api/app/services/audit.py) over them; expect zero mismatches and monotonic entry_seq per organisation. From R2, walk the audit_chain and confirm prev_hash and entry_hash link.
Recompute signature validity: for every signatures row, recompute record_hash from the referenced protocol, report or requirement-set version and compare; records that were superseded after signing must still validate against their own version.
Spot-check documents: for a sample of documents rows compare sha256 with the restored object.
Record the outcome (date, backup id, counts, mismatches, duration to a usable API) in the validation package and in the deployment record; destroy the scratch cluster and bucket.

Restore for real

Declare the incident and freeze releases; quiesce the interactive lane (release.sh --agent path) so no turn is half-written.
Restore the cluster to the chosen point in time; restore bucket objects to the same point.
Run alembic check against the restored database with the owner DSN; run make dsn-check with the application DSN.
Recompute hashes on the restored audit rows (step 3 above) before opening traffic; any mismatch is investigated first.
Roll the api, worker and web to the same tags they ran before; verify GET /api/v1/health/ready, the SSE state event on a test thread, and a signature page reading “Signature valid”.
Record the restore in the deployment record and in the change-control log of the validation package.

Evidence in the light validation package

The package (URS covering approval workflow, SOP binding, audit trail, permissions, signatures, connector reads and backup/restore; risk assessment; traceability; test evidence; release notes; change control) takes from this runbook the quarterly restore test record with its hash and signature recomputation results.

Source: PRODUCT-PLAN.md Part 4 section 8 (backup and DR), PRODUCT-CONTRACT.md section 10.7, api/app/services/audit.py (verify_row_hash), api/app/services/signatures.py (record_hash, is_valid), api/Makefile (dsn-check, migrate-check), infra/RUNBOOK-first-release.md, decisions/0014-alcoa-envelope-and-append-only-audit.md, decisions/0017-pilot-scope-validation-and-status-transitions.md