Webhook Delivery skill for Codex

A webhook platform is trustworthy when no committed event is ever lost, every request is signed in a format receivers can verify with off-the-shelf libraries, one slow customer cannot delay everyone else, and customers can see and replay what was sent.

Build or buy

Situation	Choice
Few event types, few customers, team owns a queue already	Build with the design below
Many customers, compliance needs, self-serve endpoint portal wanted	Hosted or open-source dispatcher (Svix, Hookdeck Outpost, Convoy)
Internal service-to-service events	Not webhooks: use a message broker (see rabbitmq-messaging)

Workflow

- [ ] 1. Define the event envelope and catalog
- [ ] 2. Write events to an outbox in the business transaction
- [ ] 3. Fan out to per-endpoint deliveries
- [ ] 4. Sign and send with strict timeouts
- [ ] 5. Retry, dead-letter and auto-disable
- [ ] 6. Protect the network edge (SSRF)
- [ ] 7. Give customers logs, test events and replay

1. Envelope and catalog

{
  "id": "evt_01JF3Q8Z5K7M2N",
  "type": "invoice.paid",
  "created_at": "2026-03-01T12:00:00.123Z",
  "api_version": "2026-01-15",
  "data": { "object": { "id": "inv_123", "status": "paid", "amount_paid": 4200 } }
}

• Event names are resource.past_tense_verb, documented with a JSON Schema per type.
• id is stable across retries; receivers dedupe on it.
• Include the object’s version or updated_at so receivers can discard stale events.
• Fat payloads (full object) save receivers an API call; thin payloads (ID only) avoid leaking data to misconfigured endpoints. Pick one per API and document it.
• Version payloads with the account’s pinned API version (see api-versioning).

2. Outbox

Write the event in the same database transaction as the state change, then dispatch from the outbox. Publishing directly to a queue after commit loses events when the process dies between commit and publish (see outbox-pattern).

3. Fan out

CREATE TABLE webhook_deliveries (
  id              bigserial PRIMARY KEY,
  event_id        text        NOT NULL,
  endpoint_id     bigint      NOT NULL REFERENCES webhook_endpoints(id),
  status          text        NOT NULL DEFAULT 'pending', -- pending|succeeded|failed
  attempt         int         NOT NULL DEFAULT 0,
  next_attempt_at timestamptz NOT NULL DEFAULT now(),
  last_status     int,
  last_error      text,
  UNIQUE (event_id, endpoint_id)
);
CREATE INDEX webhook_deliveries_due ON webhook_deliveries (next_attempt_at) WHERE status = 'pending';

Workers claim due rows without blocking each other:

-- the new next_attempt_at is a lease: if the worker dies, the row becomes due again
UPDATE webhook_deliveries d
SET attempt = attempt + 1, next_attempt_at = now() + interval '2 minutes'
WHERE id IN (
  SELECT id FROM webhook_deliveries
  WHERE status = 'pending' AND next_attempt_at <= now()
  ORDER BY next_attempt_at
  FOR UPDATE SKIP LOCKED
  LIMIT 50
)
RETURNING d.*;

Fairness: cap in-flight requests per endpoint (for example 5) so a slow endpoint queues its own work instead of occupying every worker.

4. Sign and send

Follow the Standard Webhooks spec so receivers can use existing verifier libraries: headers webhook-id, webhook-timestamp (Unix seconds), and webhook-signature containing v1,<base64 HMAC-SHA256 of "{id}.{timestamp}.{body}">.

import { createHmac, randomBytes } from 'node:crypto';

export const newSecret = () => `whsec_${randomBytes(32).toString('base64')}`;

export function signatures(secrets: string[], id: string, ts: number, body: string): string {
  return secrets
    .map((s) => Buffer.from(s.replace(/^whsec_/, ''), 'base64'))
    .map((key) => 'v1,' + createHmac('sha256', key).update(`${id}.${ts}.${body}`).digest('base64'))
    .join(' ');
}

export async function deliver(endpoint: Endpoint, event: Envelope) {
  const body = JSON.stringify(event);         // sign exactly the bytes you send
  const ts = Math.floor(Date.now() / 1000);
  const res = await fetch(endpoint.url, {
    method: 'POST',
    redirect: 'manual',
    signal: AbortSignal.timeout(15_000),
    headers: {
      'content-type': 'application/json',
      'user-agent': 'ExampleWebhooks/1.0',
      'webhook-id': event.id,
      'webhook-timestamp': String(ts),
      'webhook-signature': signatures(endpoint.activeSecrets, event.id, ts, body),
    },
    body,
  });
  return { ok: res.status >= 200 && res.status < 300, status: res.status };
}

• Success is any 2xx within the timeout (10-15 s). Redirects are failures; do not follow them (they bypass URL validation).
• Store response status, latency and the first few KB of the response body for logs; never store or log the secret.
• Rotation: during a rotation window (for example 24 h) sign with both old and new secrets; the header carries both signatures separated by a space.

5. Retries, dead letters, disabling

Retry on network errors, timeouts, 429 and 5xx. Treat 4xx other than 408/429 as failures that still retry (receivers often return 400 during their own deploys), but log them distinctly.

Attempt	Delay after previous (plus up to 20% jitter)
1	immediate
2	5 s
3	5 min
4	30 min
5	2 h
6	5 h
7	10 h
8	10 h

• Honor Retry-After from 429/503 if it is within the schedule.
• After the last attempt mark the delivery failed (the dead-letter state), keep it for manual replay, and notify the account owner.
• Auto-disable an endpoint after sustained failure (for example every delivery failing for 5 days), with an email warning beforehand. Sending customer emails is a product decision; confirm wording and triggers with the user.

6. SSRF protection

Customer-supplied URLs let anyone make your servers call internal addresses.

• Accept only https:// URLs on port 443 (allow http only in test mode).
• Resolve DNS at send time and reject private, loopback, link-local and metadata ranges (10/8, 172.16/12, 192.168/16, 127/8, 169.254/16, ::1, fc00::/7, fe80::/10), then connect to the resolved IP so a second lookup cannot rebind to an internal one.
• Send from an isolated egress (separate network segment or an egress proxy such as Smokescreen) with no access to internal services.
• Details and test cases live in ssrf-prevention.

7. Customer-facing tooling

• Delivery log per endpoint: event, attempts, status codes, latency, next retry.
• “Send test event” and “Resend” buttons (replay uses the same event ID so receivers dedupe correctly).
• Endpoint settings: subscribed event types, secret reveal and rotate, enable/disable.
• Publish verification snippets for common languages, linking to Standard Webhooks libraries.

Gotchas

• Signing a JS object and then sending a differently serialized string produces signatures nobody can verify; serialize once.
• Retrying synchronously inside the request that created the event couples your API latency to customers’ servers.
• One shared FIFO queue lets a single endpoint that times out at 15 s stall all deliveries.
• Promising ordering is a trap; document “at least once, unordered” and include versions.
• Event IDs regenerated on retry make receiver deduplication impossible.

Verification

• Verify a generated request with an independent Standard Webhooks verifier library (or the receiver skill’s checker script) using the endpoint secret.
• Integration test with a local HTTP server that fails twice then succeeds: three attempts, one success, delays follow the schedule (use a fake clock).
• Kill the worker mid-batch: no delivery is lost and none is sent more than the retry policy allows beyond at-least-once.
• SSRF tests: endpoints resolving to 127.0.0.1, 169.254.169.254 and a private IP are rejected at registration and at send time.

Output template

## Webhook delivery design
- Envelope: <fields>, payload style <fat|thin>
- Source of events: outbox table <name>
- Queue/claiming: <mechanism>, per-endpoint concurrency <n>
- Signing: Standard Webhooks v1, rotation window <h>
- Timeout / retry schedule / dead-letter: <values>
- Auto-disable rule: <rule>
- SSRF controls: <list>
- Customer tooling: <logs, test, replay>