Backup and Disaster Recovery skill for Codex
A backup counts once it has been restored. Codex inventories every dataset and its real backup, asks owners for RPO and RTO instead of guessing, and designs tiers with point-in-time recovery, an off-site copy and an immutable copy production credentials cannot delete. It adds freshness alerts and runs restore drills into isolated environments using a bundled drill guide that times each phase and checks the data itself.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- backup-disaster-recoveryInstalls to ~/.agents/skills/backup-disaster-recovery/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install backup-disaster-recoveryThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project backup-disaster-recoveryInstalls to .agents/skills/backup-disaster-recovery/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/backup-disaster-recovery.zipunzip -o backup-disaster-recovery.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) backup-disaster-recoveryPowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $backup-disaster-recovery, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Designs backup and disaster recovery from business RPO and RTO targets - 3-2-1 copies with an immutable or offline copy, restic and Borg for files, pgBackRest, WAL-G and managed snapshots for point-in-time database recovery, S3 Object Lock, cross-account and cross-region copies, monitoring of backup freshness, and scheduled restore drills with timed results. Use when setting up backups, auditing whether existing backups can actually be restored, defining RPO/RTO, preparing a DR runbook, or protecting against ransomware and accidental deletion.
Use it when
- A system has no backups or untested ones
- Leadership asks for RPO, RTO or a DR plan
- Ransomware or accidental deletion is a concern
- A restore drill needs planning, running or a written report
Use something else when
- You need PostgreSQL-specific pg_dump, pgBackRest or PITR commands in depth: use postgres-backup-restore
- You are handling an active outage rather than preparing for one: use incident-command
How to check the result
Each tier 0 and 1 dataset has a drill report showing a restore into an isolated environment within its RTO, with row counts and latest timestamps checked against the RPO.
The mistake it prevents
Treating a replica or a successful backup job as proof of recoverability, when replicas copy deletions instantly and jobs can succeed while skipping data.
A backup counts only once it has been restored. Done well means: every critical dataset has a written RPO and RTO agreed with its owner, at least one copy lives outside the production account and cannot be deleted by production credentials, backup freshness is monitored, and a timed restore drill has met the RTO within the last quarter.
Inspect first
- Inventory data stores: databases, object storage, file volumes, secrets and key material, SaaS data, infrastructure state (Terraform state, cluster manifests).
- For each, find the current backup: tool, schedule, retention, location, encryption, who can delete it, and the date of the last successful restore test.
- Read
AGENTS.md, infra code and cron or systemd timers to see what actually runs, not what the wiki says. - Ask the owner for RPO (how much data loss is tolerable) and RTO (how long down is tolerable). Do not invent business targets; propose tiers and let them choose.
Targets drive design
| Tier | Example | RPO | RTO | Design |
|---|---|---|---|---|
| 0 | Payments DB | under 1 min | under 15 min | Replica in another zone or region with automated failover, plus PITR |
| 1 | Main app DB | 5 min | 1 h | Continuous WAL/binlog archiving (PITR) + daily full |
| 2 | User uploads | 24 h | 4 h | Versioned bucket + daily cross-account replication |
| 3 | Internal tools | 24 h | 1 day | Nightly snapshot |
Replication is not backup: it copies DROP TABLE and ransomware encryption instantly.
Every tier needs point-in-time or versioned copies in addition to replicas.
The 3-2-1-1-0 rule
- 3 copies of the data, on 2 different media or services, 1 off-site (other region or provider), 1 immutable or offline, 0 errors in the last restore verification.
- Immutable means production credentials cannot delete or shorten retention: S3 Object Lock in compliance mode, Azure immutable blob policies, GCS bucket lock, AWS Backup Vault Lock, or a separate account whose credentials production never holds.
Tools
| Data | Default |
|---|---|
| PostgreSQL self-managed | pgBackRest or WAL-G: full + incremental + WAL archive for PITR |
| PostgreSQL / MySQL managed (RDS, Cloud SQL, Azure) | Automated backups with PITR enabled + snapshot copies to a second account/region |
| MySQL self-managed | Percona XtraBackup + binlog archiving |
| Files and volumes | restic or Borg (deduplicated, encrypted) |
| Object storage | Versioning + lifecycle + replication to a locked bucket in another account |
| Kubernetes resources and PVs | Velero, plus GitOps repo as the source of truth for manifests |
| Many AWS services | AWS Backup with cross-account copy and Vault Lock |
restic example
export RESTIC_REPOSITORY=s3:s3.amazonaws.com/acme-backups/app01
export RESTIC_PASSWORD_FILE=/etc/restic/password # 0600, also stored in the password manager
restic init # once
restic backup /srv/data --exclude-caches --tag nightly
restic forget --keep-daily 7 --keep-weekly 4 --keep-monthly 12 --prune
restic check --read-data-subset=5% # verify a sample of pack data
restic snapshots --latest 3
restic restore latest --target /tmp/restore-test --include /srv/data/important- Lose the repository password and the backups are unrecoverable. Store it in two places that do not depend on the backed-up system.
forget --prunedeletes data; when changing retention flags, run with--dry-runfirst and get the user’s confirmation.- Database files copied while the DB runs are inconsistent. Dump (
pg_dump -Fc) or use the DB’s backup tool, then back up the dump.
PostgreSQL PITR essentials
archive_mode = on,archive_command(or pgBackRest’sarchive-push) shipping WAL to object storage; alert ifpg_stat_archiver.failed_countgrows orlast_archived_timeis older than the RPO.- Full backup weekly, differential or incremental daily, WAL continuous.
- Restore to a time just before an incident with a recovery target
(
--type=time --target="2026-03-01 14:02:00+00"in pgBackRest). - Details belong to
postgres-backup-restore; this skill sets the policy and drills.
Monitoring
- Alert on freshness, not just job failure: “newest successful backup older than RPO x 1.5”. A silently skipped job never fails.
- Track backup size trend; a sudden drop often means a broken include path.
- Push a metric or heartbeat at the end of each job (Prometheus Pushgateway, healthchecks ping, CloudWatch metric) and alert when it goes missing.
Restore drills
Run quarterly for tier 0 and 1, twice a year for others, and after any major infra change. Follow references/restore-drill.md for the drill procedure, data checks and the report template; read it when planning or running a drill.
Short version:
- Restore into an isolated environment, never over production
- Time every phase: locate backup, provision, transfer, restore, verify, cut over
- Verify data, not just exit codes: row counts, latest timestamps vs RPO, checksums on sample files, application smoke test
- Record measured RPO and RTO and compare with targets
- File follow-ups for every manual step that was not in the runbook
DR runbook contents
- Decision criteria and who declares a disaster.
- Ordered recovery steps per tier, with exact commands and where credentials are found when the primary SSO or password manager is also down (break-glass accounts).
- Dependencies order: network and DNS, secrets, databases, then apps.
- Communication plan and status page owner.
- How to fail back afterwards.
Safety
Restores over existing data, deleting backups or snapshots, changing retention or lock settings, and failing over production are destructive or outward-facing. Prepare commands and wait for the user’s explicit confirmation before running any of them, and always restore to a new target first.
Verification
- Inventory table complete, each row with tool, location, immutability and last tested restore.
- Freshness alert fires in a test (pause a job in staging).
- A drill has restored each tier 0 and 1 dataset within its RTO, with data checks passed.
Output template
## Backup and DR review: <system>
| Dataset | Tier | RPO / RTO target | Backup | Off-site | Immutable | Last restore test | Measured RTO |
|---|---|---|---|---|---|---|---|
| orders-db | 1 | 5 min / 1 h | pgBackRest + WAL to S3 | eu-west-2 copy | Object Lock 35 d | 2026-09-14 | 42 min |
Gaps: <untested datasets, single-account storage, missing freshness alerts>
Actions needing approval: <retention changes, deletions, failover tests>What the zip contains
Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "Backup and Disaster Recovery"
short_description: "A backup counts once it has been restored"
default_prompt: "$backup-disaster-recovery Audit our backups and tell me which ones we have never actually restored"references/restore-drill.md
# Restore drill procedure
Use this when planning or running a restore test. The goal is a measured RPO and RTO and a
runbook that a tired on-call engineer can follow without the author present.
## Before the drill
- Pick the scenario. Rotate between them across drills:
- Accidental deletion: restore one table, bucket prefix or directory to a point in time.
- Full loss of a database or volume: rebuild from backups only.
- Region or account loss: restore in a different region or account using only the
off-site copy and break-glass credentials.
- Ransomware: assume production credentials are compromised; restore from the
immutable copy without using them.
- Choose the person running it: ideally not the one who built the backups.
- Prepare an isolated target (separate VPC, namespace or account) with no route to
production writes. Confirm that application config in the restored environment cannot
send email, charge cards or call production webhooks.
- Write down the target point in time and the expected data at that point (for example,
the newest order ID created before 14:00 UTC).
- Tell stakeholders the window; a restore can load shared storage or networks.
## Timeline to record
| Phase | Start | End | Duration | Notes |
|---|---|---|---|---|
| Declare and locate backup | | | | Which backup, how found |
| Obtain credentials and access | | | | Break-glass path worked? |
| Provision target | | | | |
| Transfer data | | | | Throughput MB/s |
| Restore and replay logs | | | | |
| Verify data | | | | |
| Application smoke test | | | | |
| Total (RTO) | | | | Compare with target |
Measured RPO = target point in time minus timestamp of the newest data actually present.
## Data verification checklist
- Row counts for the largest and most important tables compared with a production
snapshot taken at the same point in time (allow for writes after the target).
- Newest timestamp in key tables (`max(created_at)`) within the RPO of the target time.
- Referential integrity spot checks: orphaned foreign keys, missing parent rows.
- Checksums (`sha256sum`) of a random sample of restored files against the source.
- Database consistency check where available (`pg_amcheck`, `mysqlcheck`, `DBCC CHECKDB`).
- Application smoke test: log in, read a recent record, run a read-only report.
- Encryption: confirm the restore did not require a key that exists only in the lost system.
## Common findings
- Backups encrypted with a key stored on the server that was lost.
- Restore needs a tool version no longer installed anywhere.
- Transfer from cold storage tiers (Glacier Deep Archive and similar) takes hours to
start; RTO assumptions ignored retrieval time.
- Runbook references a person, wiki or SSO that is unavailable in the scenario.
- Backup job succeeded but excluded a newly added directory or schema.
- Point-in-time recovery fails because a WAL or binlog segment is missing in the chain.
## Drill report template
```markdown
## Restore drill: <dataset>, <date>
Scenario: <accidental deletion | full loss | region loss | ransomware>
Operator: <name> Observer: <name>
Target point in time: <timestamp> Backup used: <id and location>
Results
- RTO target <x>, measured <y> -> pass | fail
- RPO target <x>, measured <y> -> pass | fail
- Data checks: <passed / failed items>
Problems encountered
1. <problem> -> follow-up <ticket>, owner, due date
Runbook changes made
- <step added or corrected>
Next drill: <date and scenario>
```
## After the drill
- Tear down the isolated environment, including restored copies of personal data, once
the user confirms the report is saved.
- Update the inventory table with the test date and measured RTO.
- Fix runbook gaps immediately while details are fresh.
About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.