Hotfix Process skill for Codex

A hotfix succeeds when users stop being hurt as fast as possible, the shipped change is the smallest one that fixes the problem, it was tested on the code that is actually in production, and the fix also lands on every other branch so the bug does not return in the next release. Speed comes from a rehearsed procedure, not from skipping checks.

First: is a hotfix the right tool?

Mitigation usually beats a code change. Try these in order:

Option	Time to relief	Use when
Turn off a feature flag / kill switch	Seconds	The bug is behind a flag
Roll back to the previous release	Minutes	The bug arrived in the latest deploy and its migrations are backward compatible
Config or infrastructure change (scale, rate limit, block a route)	Minutes	Load, a bad client, or a dependency outage
Hotfix (code change)	30 min to hours	None of the above work, or rollback would lose data or reintroduce a worse bug

Ask the user which mitigation is acceptable before touching production. Every production action (rollback, flag change, deploy) needs explicit confirmation from them.

Workflow

- [ ] 1. Confirm impact and the version actually running in production
- [ ] 2. Mitigate if possible (flag, rollback, config); record the time
- [ ] 3. Reproduce against the deployed version, not main
- [ ] 4. Branch from the deployed tag
- [ ] 5. Write a failing regression test, then the minimal fix
- [ ] 6. Expedited review by one more person; CI green
- [ ] 7. Tag patch version; deploy with staged rollout (user confirms)
- [ ] 8. Verify in production with the metric that showed the bug
- [ ] 9. Port the fix to main and other supported branches
- [ ] 10. Record follow-ups and hand off to the postmortem

Step 1: what is running?

Never assume main is what is deployed.

git fetch --tags origin
git describe --tags --abbrev=0 origin/main        # latest tag on main (may not be deployed)
# Ask the deployment system for the truth: image tag, release name, /version endpoint
curl -s https://api.example.com/version            # if the service exposes one (needs network)

Step 4: branch from the deployed tag

git switch -c hotfix/2.4.1 v2.4.0                 # the exact deployed version

If the project uses release branches, branch from release/2.4 instead and confirm its tip equals the deployed tag (git rev-parse release/2.4 v2.4.0^{commit}). Branching from main would ship every unreleased change along with the fix.

Step 5: minimal patch

• Write the regression test first and watch it fail on the deployed code. This is the proof that you fixed the reported bug and not a neighbouring one.
• Change as few lines as possible. No refactors, no renames, no dependency bumps, no formatting changes, no “while I’m here” fixes. Each extra line is untested risk.
• Prefer a guard or a revert of the offending commit over a redesign:

git revert --no-edit <bad-commit-sha>      # often the safest hotfix of all

• If a database change is needed, it must be additive and safe for both old and new code. Destructive data repair is a separate, reviewed step with a backup, never bundled.
• If the proper fix is large, ship a narrow mitigation now (guard, validation, disabled path) and schedule the real fix through the normal process.

Step 6: expedited review, not no review

• One reviewer who knows the area; share the failing test, the diff and the incident link.
• Full required CI on the hotfix branch. If a slow suite must be skipped, the user decides that explicitly and it runs right after deploy.
• Keep the PR description short: impact, root cause in one sentence, the fix, how it was tested, rollback plan (usually: redeploy v2.4.0).

Step 7: release and deploy

git tag -a v2.4.1 -m "Hotfix: prevent double charge on payment retry timeout"
git push origin hotfix/2.4.1 v2.4.1                     # confirm with user first
gh release create v2.4.1 --verify-tag --notes-file HOTFIX_NOTES.md

Deploy with the fastest staged rollout the platform supports (canary, then all). Watch the signal that detected the incident, plus error rate and latency, at each stage.

Step 8: verify in production

The fix is verified when the symptom metric returns to baseline and a direct check of the failing case passes (a synthetic request, a replayed job, a support-reported account). “Deploy succeeded” is not verification.

Step 9: port everywhere

Missed ports are why the same bug reappears a release later.

git switch main && git pull
git cherry-pick -x <hotfix-commit-sha>           # forward-port to main via a PR
git switch release/2.3
git cherry-pick -x <hotfix-commit-sha>           # backport to other supported lines
git cherry -v main hotfix/2.4.1                   # '+' lines are commits not yet on main

Open a PR for each target branch. If main has diverged so the patch does not apply, rewrite the fix for main with the same regression test, and reference the hotfix commit. Delete the hotfix branch after the PRs merge, if the team’s conventions allow.

Gotchas

• Hotfixing on top of main silently ships unreleased features and migrations.
• A fix verified only on staging, when staging runs main, proves nothing about the deployed version.
• Bundling a “small” refactor into the hotfix makes it impossible to roll back the fix without losing the refactor, and doubles the review surface.
• Patch version reuse: if v2.4.1 was tagged and pulled, the next attempt is v2.4.2. Never move a published tag.
• Forgetting to update the changelog leaves users unable to tell whether they are on the fixed version.
• Messages to customers or status pages are outward-facing; draft them for the user to send rather than posting them.

Output

## Hotfix v2.4.1
**Incident:** <link>  **Impact:** duplicate charges on payment retry, ~0.3% of checkouts since 09:12 UTC
**Mitigation:** retries disabled via `payments-retry-enabled` flag at 09:40 (confirmed by user)
**Root cause (short):** retry after timeout reused no idempotency key (commit abc123 in v2.4.0)
**Fix:** pass existing idempotency key on retry (+6/-1 lines), regression test test_retry_idempotent
**Branch / tag:** hotfix/2.4.1 from v2.4.0 -> v2.4.1
**Verified:** duplicate-charge alert clear 30 min; synthetic retry test passes in prod
**Ported:** main (#912), release/2.3 (#913)
**Follow-ups:** re-enable flag after 24 h; postmortem owner <name>

Verification

• The regression test fails on the deployed tag and passes on the hotfix.
• Production symptom metric is back to baseline after rollout.
• git cherry -v shows the fix present on main and every supported release branch.