Distributed Locks skill for Codex

Done well means the team knows which kind of lock it has: an efficiency lock (duplicate work is wasteful but harmless) or a correctness lock (duplicate work corrupts data). An efficiency lock can be a simple lease. A correctness lock needs a fencing token checked by the resource being protected, because any lease can expire while its holder is paused. Often the best lock is none: a database constraint or conditional write does the job.

Before adding a lock

1. Read AGENTS.md and existing coordination code: rg -n "advisory_lock|SETNX|NX.*PX|Redlock|redlock|\.lock\(|Lease|leaderelection|SKIP LOCKED" src.
2. Write the failure scenario in one sentence: “Two pods run the nightly invoice job and customers get billed twice.” That sentence decides everything below.
3. Identify the protected resource (a row, a file, an external API) and whether it can check a token or condition itself.

Do you need a lock?

Problem	Better than a lock
Prevent duplicate rows (two signups, two charges for one order)	Unique constraint or idempotency key; handle the conflict
Lost updates on the same row	Optimistic concurrency: UPDATE ... SET version = version + 1 WHERE id = $1 AND version = $2, retry on 0 rows
Many workers pulling jobs from a table	SELECT ... FOR UPDATE SKIP LOCKED LIMIT n inside a transaction
All events for one entity must be processed in order by one worker	Partition the queue or stream by entity id (Kafka key, SQS FIFO group id)
Counter or balance changes	Atomic increments or conditional writes in the store itself
Exactly one scheduler instance	Leader election with a lease, or a platform scheduler (Kubernetes CronJob with concurrencyPolicy: Forbid)

Use a distributed lock only when the work spans systems that cannot express the condition themselves and serialisation is truly required.

Lock types and where to get them

Need	Recommended	Fencing token available
Efficiency lock, Redis already in the stack	Redis SET key token NX PX ttl with compare-and-delete release (or the client library’s lock)	No (not safely)
Lock around work inside one Postgres database	Transaction-scoped advisory lock pg_try_advisory_xact_lock	The transaction itself is the boundary
Correctness lock across services	etcd or ZooKeeper (consensus-backed)	Yes: the lock key’s revision or the znode’s version
AWS-native, no extra infrastructure	DynamoDB conditional writes on a lock item with lease expiry and a counter	Yes, if you increment a counter attribute
Leader election in Kubernetes	coordination.k8s.io Lease via client-go leaderelection or controller-runtime	Lease transitions count; still pair with fencing for external writes

Redlock (locking across several independent Redis nodes) does not provide fencing and relies on timing assumptions; do not use it as a correctness lock. If someone proposes it, ask which failure the lock must prevent and route to the table above.

Implementations

Redis lease (efficiency only)

from redis.exceptions import LockError

lock = redis_client.lock("lock:report:2026-10-28", timeout=60)  # redis-py Lock
if not lock.acquire(blocking=False):
    return  # someone else is doing it
try:
    build_report()          # must finish well within 60 s, or extend the lease
finally:
    try:
        lock.release()      # token-checked release; raises if the lease was lost
    except LockError:
        log.warning("lease expired before release; work may have overlapped")

Hand-rolled equivalent: acquire with SET lock:x <random-token> NX PX 60000, release with a Lua script that deletes only if the stored value equals your token:

if redis.call("GET", KEYS[1]) == ARGV[1] then
  return redis.call("DEL", KEYS[1])
end
return 0

Never release with a plain DEL: if your lease expired and another worker acquired the lock, you delete their lock.

Postgres advisory lock

BEGIN;
SELECT pg_try_advisory_xact_lock(hashtextextended('invoice-run:2026-10', 0)) AS got;
-- if got is false: ROLLBACK and exit
-- do the work in this same transaction; the lock releases on COMMIT or ROLLBACK
COMMIT;

Prefer the _xact_ variants. Session-level advisory locks leak when a pooled connection is returned without unlocking, and they do not work through PgBouncer in transaction pooling mode because consecutive statements may run on different server connections.

Fencing tokens (correctness)

1. Acquire the lock from a store that returns a monotonically increasing number with each grant (etcd revision, ZooKeeper zxid or version, a DynamoDB counter attribute updated in the same conditional write).
2. Pass the token with every write to the protected resource.
3. The resource rejects writes carrying a token lower than the highest it has seen:

UPDATE ledgers
SET balance = $1, fence = $2
WHERE id = $3 AND fence < $2;   -- 0 rows updated means a newer holder exists: abort

If the resource cannot check a token (a third-party API), you cannot get correctness from a lock alone; use idempotency keys on that API instead.

Leases and long work

• Lease TTL must exceed the worst-case pause (GC, VM stall, network partition) plus work time, or the holder must renew. Renew at about one third of the TTL and stop working immediately if renewal fails.
• Check the lease (or carry the fence) before each side effect in long jobs, not only at the start.

Gotchas

• Clock assumptions: lease expiry measured on different machines drifts; a paused process does not know its lease expired. This is why fencing exists.
• Holding a lock across a network call to a slow dependency turns the dependency’s latency into contention for everyone.
• Lock keys built from user input without normalisation (case, whitespace) let two “different” keys protect the same resource.
• hashtext on lock names can collide; use hashtextextended (64-bit) or a lookup table of integer lock ids for important locks.
• Cron jobs on every replica plus a lock is fragile; a single scheduler (CronJob, cloud scheduler) plus idempotent jobs is simpler.
• Deleting lock keys or terminating backends that hold advisory locks in production to “unstick” a job can cause the duplicate work the lock prevented; confirm with the user that the holder is truly dead before forcing a release.

Verification

• Concurrency test: start N workers (threads or processes, N of at least 10) against a real Redis or Postgres in a container and assert the protected action ran exactly once, or that duplicates were rejected by the constraint or fence.
• Expiry test: make the holder sleep past its TTL, let a second worker acquire, then let the first resume. With fencing, the first worker’s write must be rejected. Without fencing, document that overlap is possible and harmless (efficiency lock).
• Release test: the token-checked release must not delete another holder’s lock.

Output template

## Coordination design: <operation>
**Failure being prevented:** <one sentence>
**Lock type:** efficiency | correctness
**Mechanism:** <constraint / optimistic version / SKIP LOCKED / lease + fence / leader election>
**Store:** <Redis / Postgres / etcd / DynamoDB / Kubernetes Lease>
**TTL and renewal:** <values, renewal interval, behaviour on renewal failure>
**Fencing:** <token source, where it is checked>, or "not needed because ..."
**Tests:** <concurrency, expiry, release tests and results>