Event-Driven Architecture skill for Codex

An event-driven design is working when every event has one owner and a versioned schema, consumers produce the same result whether a message arrives once or five times, ordering is guaranteed exactly where the business needs it and nowhere else, and a failed message ends up somewhere a human can see and replay.

Decide whether events fit

Need	Events?
Several independent reactions to one fact (email, analytics, search index)	Yes, publish/subscribe
Absorb bursts, decouple producer availability from consumer	Yes, queue
Caller needs an answer now to continue (price, auth decision)	No, synchronous call
Multi-step business transaction across services	Events plus a saga ($saga-pattern)
Two modules in one deployable	In-process events or direct calls first ($modular-monolith)

Kinds of messages

Kind	Example	Payload	Coupling
Event notification	OrderPlaced {orderId}	IDs only; consumers call back for details	Low payload, runtime coupling via callbacks
Event-carried state transfer	OrderPlaced {orderId, lines, total, customer}	Everything consumers need	No callbacks; schema coupling
Command (message)	SendInvoice {orderId}	Intent to one handler	Directed; not pub/sub
Domain event (internal)	Raised inside an aggregate	Domain types	Private to the service

Default for integration events: event-carried state transfer with the fields current consumers need, so a slow or down producer does not break consumers. Name events in past tense business language (PaymentCaptured), never CRUD (PaymentUpdated) and never commands disguised as events.

Envelope and schema

Use the CloudEvents attributes even if you do not use a CloudEvents SDK:

{
  "specversion": "1.0",
  "id": "5f0c2a8e-3b1f-4c7e-9a51-2f8d1e6c0b44",
  "source": "/services/orders",
  "type": "com.acme.orders.order_placed.v1",
  "subject": "order/ord_8412",
  "time": "2026-03-14T10:22:31Z",
  "datacontenttype": "application/json",
  "traceparent": "00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01",
  "data": { "orderId": "ord_8412", "customerId": "cus_19", "total": { "amount": "84.10", "currency": "EUR" } }
}

• id is unique per event and stable across redeliveries; consumers dedupe on it.
• traceparent (W3C Trace Context, also the CloudEvents distributed tracing extension) links async hops in traces.
• Money as string decimals plus currency; timestamps in UTC ISO 8601.

Schema evolution rules (Avro, Protobuf or JSON Schema, checked by a schema registry or a CI compatibility test):

Change	Safe?
Add optional field with default	Yes (backward compatible)
Remove optional field consumers do not use	Yes after checking consumers
Rename field, change type, make optional required	No: publish ...v2 alongside v1, migrate consumers, retire v1
Change meaning of a field without changing its name	Never

Set registry compatibility to BACKWARD (or BACKWARD_TRANSITIVE for long retention topics). Document contracts in an AsyncAPI 3 file next to the producer.

Broker choice

Broker	Strengths	Pick when
Kafka (or Redpanda, MSK, Confluent)	Ordered partitions, retention and replay, high throughput	Event streams many consumers read at their own pace; replay matters
RabbitMQ	Flexible routing, per-message acks, priorities	Work queues and routing-heavy command messaging
SQS + SNS	Fully managed, simple, FIFO option with group IDs	AWS shop, moderate throughput, no replay need
EventBridge	Managed routing by rules, SaaS sources	AWS integration events between teams/accounts
Google Pub/Sub	Managed, ordering keys, push or pull	GCP shop
NATS JetStream	Light, fast, simple ops	Edge, IoT, low-ops streaming

Default: the managed queue of your cloud for work distribution; Kafka only when you need replay, long retention or very high throughput and can operate it.

Ordering

• Brokers order only within a partition, FIFO group or ordering key. Choose the key as the entity whose events must be ordered (orderId), never a random value.
• Global ordering does not scale and is rarely needed. Ask which entity needs order.
• Consumers must still tolerate reordering across keys and redelivery: compare an entity version or sequence in the payload and ignore stale updates.
• Hot keys (one huge tenant) bottleneck a single partition; consider a compound key if per-tenant ordering is not required.

Delivery and idempotency

Assume at-least-once delivery everywhere. “Exactly-once” broker features (Kafka transactions) cover read-process-write within Kafka only; side effects in databases, emails or APIs still need idempotent handling.

CREATE TABLE processed_messages (
  consumer   text NOT NULL,
  message_id text NOT NULL,
  processed_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (consumer, message_id)
);

def handle(event: dict) -> None:
    with db.transaction() as tx:
        inserted = tx.execute(
            "INSERT INTO processed_messages (consumer, message_id) VALUES (%s, %s) "
            "ON CONFLICT DO NOTHING",
            ("billing", event["id"]),
        ).rowcount
        if inserted == 0:
            return                      # duplicate: already applied
        apply_business_change(tx, event["data"])
    # commit/ack the message only after the transaction commits

Publishing reliably from a service that also writes to its database needs the outbox, not “write then publish” ($outbox-pattern).

Kafka producer settings worth checking: acks=all, enable.idempotence=true (default in modern clients), and a min.insync.replicas of 2 on topics with replication factor 3. Consumers: commit offsets after processing, not before.

Failure handling

attempt -> transient error -> retry with exponential backoff + jitter (bounded, e.g. 5 tries)
        -> permanent error (validation, unknown schema) -> dead-letter immediately
        -> retries exhausted -> dead-letter with error, attempt count, original headers

• Every queue/subscription has a DLQ with an alert on depth > 0 and a documented replay command. A DLQ nobody watches is data loss with extra steps.
• Do not block a Kafka partition forever on one poison message; route it to a retry or dead-letter topic and continue.
• Set visibility timeout (SQS) or ack deadline (Pub/Sub) above the p99 handling time.

Workflow

- [ ] 1. Read AGENTS.md and existing messaging code; list current producers and consumers
- [ ] 2. Name the event, its owner, and consumers; choose state transfer vs notification
- [ ] 3. Write the schema and envelope; register it or add a compatibility check
- [ ] 4. Choose the partition/ordering key from the entity that needs order
- [ ] 5. Publish via outbox; consume idempotently; ack after commit
- [ ] 6. Configure retries, DLQ, alerts and a replay procedure
- [ ] 7. Propagate trace context; add lag and DLQ dashboards
- [ ] 8. Document in AsyncAPI; add a contract test between producer and consumer

Gotchas

• Event chains that implement a hidden synchronous workflow (A emits, B reacts and emits, C reacts…) are hard to reason about. When the steps form one business process, make it explicit with an orchestrated saga.
• Consumers that call back to the producer for every event re-create the runtime coupling events were supposed to remove.
• Using events to replicate an entire database to other services creates a distributed monolith. Publish facts consumers need, not every column change.
• PII in events lives as long as topic retention and every consumer’s copy. Minimise it and plan erasure (tokenise or encrypt per subject).
• Changing a partition count on a Kafka topic changes key-to-partition mapping and breaks per-key ordering during the transition.

Verification

• Deliver every test event twice and out of order in integration tests; state must match single in-order delivery.
• Kill a consumer mid-batch; on restart no message is lost and no side effect is doubled.
• Schema compatibility check fails CI when a field is renamed.
• A poison message reaches the DLQ, fires the alert, and the replay command restores it after a fix.

Output contract

**Event:** com.acme.orders.order_placed.v1 (owner: orders) -> consumers: billing, email, search
**Schema:** schemas/order_placed.v1.avsc, compatibility BACKWARD, AsyncAPI: docs/asyncapi.yaml
**Ordering key:** orderId | **Broker/topic:** orders.events (12 partitions, RF 3)
**Delivery:** outbox -> at-least-once; consumers dedupe on event id
**Failure path:** 5 retries with backoff -> orders.events.dlq, alert on depth > 0, replay: <cmd>