Saga Pattern skill for Codex

A saga is right when every step either completes or is undone by a defined compensation, the saga’s state survives crashes and redeploys, every participant tolerates duplicate requests, and users and support can see where a stuck saga is and why. Without those four properties, a saga is just a sequence of calls that fails halfway.

When a saga is the answer

Situation	Approach
Steps span services/APIs with separate databases	Saga
All data in one database	Local transaction; no saga
Steps can tolerate eventual consistency with compensations	Saga
Strict atomic visibility required across systems (rare)	Rethink boundaries; 2PC/XA across services is brittle and usually unavailable on cloud services
Long-running process with human steps or waits of hours/days	Saga on a durable workflow engine

Orchestration or choreography

	Orchestration	Choreography
Control	One orchestrator tells participants what to do	Participants react to each other’s events
Visibility	State in one place; easy to answer “where is order 812?”	Spread across services; needs tracing
Coupling	Orchestrator knows all steps	Each service knows the events it reacts to
Fits	4+ steps, branching, timeouts, compensations	2-3 steps, simple linear flows

Default: orchestration for anything with compensations. Choreographed sagas with compensations become an implicit state machine nobody can see.

Design the steps

Classify each step (Richardson’s taxonomy):

Type	Meaning	Rule
Compensatable	Can be undone by a compensating action	Place before the pivot
Pivot	Point of no return; after it, the saga must finish	Exactly one; often the payment capture or an external commitment
Retriable	Guaranteed to succeed eventually with retries	Place after the pivot; must be idempotent

Example, order placement:

#	Step	Type	Compensation
1	Create order (PENDING)	Compensatable	Mark order REJECTED
2	Reserve inventory	Compensatable	Release reservation
3	Authorise payment	Compensatable	Void authorisation
4	Capture payment	Pivot	- (refund is a business process, not a rollback)
5	Confirm reservation, approve order	Retriable	-
6	Request shipment	Retriable	-

Compensations are business actions, not database rollbacks: “release reservation”, “void authorisation”, “send cancellation email”. They must be idempotent and must succeed eventually (retry forever with backoff and alerting).

Implementation options

Option	Pick when
Temporal (or similar durable execution)	Complex sagas, long waits, code-first; team can run or buy Temporal
AWS Step Functions	AWS shop; visual state machine; Catch blocks route to compensations
Persisted state machine in your DB + outbox	Few sagas, no new infrastructure, moderate complexity
Choreography with events	2-3 linear steps, no compensations beyond one

Temporal (Python SDK)

from datetime import timedelta
from temporalio import workflow
from temporalio.common import RetryPolicy

with workflow.unsafe.imports_passed_through():
    from .activities import (reserve_inventory, release_inventory, authorize_payment,
                             void_authorization, capture_payment, approve_order, reject_order)

STEP = dict(start_to_close_timeout=timedelta(seconds=15),
            retry_policy=RetryPolicy(maximum_attempts=5, backoff_coefficient=2.0))
FOREVER = dict(start_to_close_timeout=timedelta(seconds=15),
               retry_policy=RetryPolicy(maximum_interval=timedelta(minutes=5)))  # unlimited attempts

@workflow.defn
class PlaceOrderSaga:
    @workflow.run
    async def run(self, order_id: str) -> str:
        compensations = []
        try:
            await workflow.execute_activity(reserve_inventory, order_id, **STEP)
            compensations.append(release_inventory)
            await workflow.execute_activity(authorize_payment, order_id, **STEP)
            compensations.append(void_authorization)
        except Exception:
            for undo in reversed(compensations):
                await workflow.execute_activity(undo, order_id, **FOREVER)
            await workflow.execute_activity(reject_order, order_id, **FOREVER)
            return "rejected"
        await workflow.execute_activity(capture_payment, order_id, **FOREVER)   # pivot
        await workflow.execute_activity(approve_order, order_id, **FOREVER)    # retriable
        return "approved"

Activities receive the same order_id on every retry and must use it as the idempotency key toward the participant (e.g. the payment provider’s idempotency key).

Persisted state machine (no workflow engine)

CREATE TABLE order_sagas (
  saga_id      uuid PRIMARY KEY,
  order_id     text NOT NULL UNIQUE,
  state        text NOT NULL,      -- STARTED, INVENTORY_RESERVED, PAYMENT_AUTHORIZED, COMPENSATING, DONE, FAILED
  step_attempt int  NOT NULL DEFAULT 0,
  next_run_at  timestamptz NOT NULL DEFAULT now(),
  last_error   text,
  version      int  NOT NULL DEFAULT 0,
  updated_at   timestamptz NOT NULL DEFAULT now()
);

• A worker picks due sagas with FOR UPDATE SKIP LOCKED, performs the next step, and updates state with an optimistic version check, all with commands sent via the outbox.
• Participant replies (events or HTTP responses) advance the state; unknown or duplicate replies are ignored based on state.
• A timeout per state moves the saga to COMPENSATING when a participant never answers.

Isolation countermeasures

Sagas lack isolation: other requests see intermediate states. Choose explicitly:

Anomaly	Countermeasure
Others act on a not-yet-final order	Semantic lock: state PENDING; other operations reject or wait on pending records
Lost update between saga and a user edit	Reread value / version check before the final step
Inventory oversold while saga runs	Reserve (decrement available) as step 2, not at the end
Reports show half-done sagas	Exclude pending states, or show them as pending

Workflow

- [ ] 1. Read AGENTS.md and existing messaging/workflow code; confirm a saga is needed
- [ ] 2. List steps; classify compensatable / pivot / retriable; define compensations
- [ ] 3. Choose orchestration engine or persisted state machine (table above)
- [ ] 4. Make every participant endpoint idempotent with the saga or order ID as key
- [ ] 5. Add per-step timeouts and the path from timeout to compensation
- [ ] 6. Add semantic lock states visible to the rest of the system
- [ ] 7. Expose saga state for support; alert on sagas stuck past their SLA
- [ ] 8. Test failure at every step, duplicate replies, and crash/restart mid-saga

Gotchas

• Compensation is not “undo the database write”. Emails were sent and webhooks fired; the compensation must be a meaningful business action (cancellation email, refund).
• A participant timing out does not mean it failed. Compensate only with idempotent actions that also handle “the original request actually succeeded later”.
• Steps after the pivot must not fail permanently. If a step after capture can be rejected by business rules, move that check before the pivot.
• Hand-rolled sagas without persisted state lose progress on deploy. If the saga lives in memory or a request thread, it is not a saga.
• Do not trigger real payments, refunds or emails while testing against production systems; use sandbox credentials and ask the user before any live-money test.

Verification

• Fault-injection tests: fail each step in turn; the system ends in a consistent final state (all compensations ran, or saga completed).
• Duplicate every participant reply and command; no double reservation or charge.
• Kill the worker or orchestrator mid-saga; it resumes from persisted state.
• A dashboard or query lists sagas by state with age; stuck ones alert.

Output contract

**Saga:** PlaceOrder (orchestrated, Temporal | Step Functions | DB state machine)
**Steps:** | # | Step | Participant | Type | Compensation | Timeout |
**Pivot:** capture payment
**Idempotency keys:** order_id per participant call
**Isolation:** semantic lock via order.status = PENDING
**Observability:** state query/dashboard <link>, alert when age > <SLA>
**Tests:** failure at each step, duplicates, restart mid-saga