Regex Crafting skill for Codex

A regex is done when it targets one named engine, is anchored the way the use case needs, passes a written table of matching and non-matching cases (including the nasty ones), runs in linear time on hostile input when the input is untrusted, and is readable by the next person through named groups, verbose mode or a comment. If a real parser exists for the format, use it instead.

First decide: regex or parser?

Input	Use instead of a regex
URLs	new URL() (JS), urllib.parse (Python), net/url (Go)
Email addresses	A permissive shape check, then a confirmation email
IP addresses, CIDR	ipaddress (Python), net.ParseIP / netip (Go), node:net isIP
Dates and times	Parse with the date library; a regex can pre-check shape only
JSON, HTML, XML, CSV	The real parser. Nested structures are not regular
Semantic versions	The language’s semver library; the official semver.org pattern if you must
Programming-language source	An AST tool (see $codemods)

Workflow

- [ ] 1. Identify the engine and the API (match vs search vs fullmatch, flags)
- [ ] 2. Write the test table first: should-match, should-not-match, edge cases
- [ ] 3. Write the simplest pattern that passes; anchor it
- [ ] 4. ReDoS review if input is untrusted or unbounded
- [ ] 5. Make it readable: named groups, verbose mode or comment
- [ ] 6. Run the table and an adversarial timing test in the target engine

1. Engine matters

Engine	Notable behaviour
JavaScript	Backtracking. Lookbehind supported. Named groups (?<n>...). u or v flag for correct Unicode; v adds set operations. No atomic groups or possessive quantifiers. A g or y regex is stateful via lastIndex
Python re	Backtracking. (?P<n>...) names. re.VERBOSE. Atomic groups (?>...) and possessive *+ since 3.11. \d, \w are Unicode-aware for str patterns. No timeout (the third-party regex module has one)
Go regexp, Rust regex, RE2	Linear time guaranteed. No backreferences, no lookaround
Java	Backtracking. Possessive and atomic supported. Pattern.compile once and reuse
.NET	Backtracking by default; pass a match timeout, or RegexOptions.NonBacktracking (.NET 7+) for linear time without lookaround
PCRE / PHP	Backtracking with possessive and atomic. PHP preg_* returns false on backtrack limit; check for it
grep / sed	BRE by default; grep -E for ERE (no \d, use [0-9]); grep -P for PCRE where available (not BSD grep)
ripgrep	Rust regex by default (linear); rg -P switches to PCRE2 for lookaround
PostgreSQL	~, ~*, regexp_match; ARE syntax; \m/\M word boundaries

Check the API, not just the pattern. Python re.match anchors only at the start; re.fullmatch anchors both ends. JS test with a g flag skips matches on repeated calls.

2. Test table first

import re
import pytest

SLUG = re.compile(r"[a-z0-9]+(?:-[a-z0-9]+)*")

@pytest.mark.parametrize("s", ["a", "abc-123", "a-b-c", "x" * 64])
def test_slug_accepts(s):
    assert SLUG.fullmatch(s)

@pytest.mark.parametrize("s", ["", "-a", "a-", "a--b", "A", "a b", "a_b", "ä", "a\n"])
def test_slug_rejects(s):
    assert not SLUG.fullmatch(s)

Always include: empty string, leading and trailing separators, doubled separators, uppercase, whitespace, a trailing newline, non-ASCII letters and digits, the maximum length, and one value that is almost valid.

3. Anchoring rules

• Validation: match the whole string. Python fullmatch; JS ^...$ without the m flag; Java matches(). In Python, $ also matches before a trailing newline: "abc\n" passes ^abc$. Use fullmatch or \Z.
• Extraction: anchor to context (\b, a preceding literal, a lookbehind) so partial tokens do not match inside larger ones.
• ^ and $ with the multiline flag mean line boundaries; use it only when you mean it.

4. ReDoS review

Backtracking engines go exponential or polynomial when one input can be matched in many ways. Patterns to flag:

Shape	Example	Fix
Nested quantifier	(a+)+, (\w+\s?)*	Flatten: \w+(?:\s\w+)*
Overlapping alternation under a quantifier	`(a	ab)*, (\d
Adjacent overlapping quantifiers	\s*\s*, .*.*=, \d+\d+	Merge them or separate with a required literal
Unbounded wildcard before a literal that may be absent	^(.*)=(.*)$ on long lines without =	Use a negated class: ^([^=]*)=(.*)$

Fixes in order of preference: rewrite so each character can be consumed only one way; use negated character classes instead of .*; bound repetition ({1,64}); cap input length before matching; use possessive or atomic groups where supported; switch to a linear engine (RE2 bindings, Rust, Go, .NET NonBacktracking); set a timeout as a last line of defence.

Adversarial timing test: feed a long string that almost matches and then fails at the end.

import re, time
pat = re.compile(r"^(\w+\s?)*$")        # vulnerable
s = "a" * 22 + "!"
t = time.perf_counter(); pat.match(s); print(f"{time.perf_counter() - t:.2f}s")
# ~0.5s at 22 chars, roughly x2.5 per extra char; ^\w+(?:\s\w+)*$ takes ms on 100k chars

Run timing tests with an outer timeout (timeout 5 python3 check.py; gtimeout from coreutils on macOS) so a vulnerable pattern cannot hang the session. For JavaScript codebases, eslint-plugin-regexp includes regexp/no-super-linear-backtracking and regexp/no-misleading-capturing-group to catch these in CI.

5. Make it readable

LOG_LINE = re.compile(
    r"""
    ^(?P<ts>\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}(?:\.\d+)?Z)   # ISO-8601 UTC timestamp
    \s+(?P<level>DEBUG|INFO|WARN|ERROR)                     # level, fixed set
    \s+\[(?P<req>[0-9a-f]{8})\]                             # request id, 8 hex chars
    \s+(?P<msg>.*)$                                         # rest of the line
    """,
    re.VERBOSE,
)

In JS, build long patterns from commented parts or keep a comment above the literal with an example of a matching line. Name groups whenever there is more than one.

Gotchas

• [A-z] includes [ \ ] ^ _ and backtick. Write [A-Za-z].
• Unicode: Python \d matches Arabic-Indic and other digits in str patterns; use [0-9] or re.ASCII for ASCII-only validation. JS without u treats astral characters (emoji) as two code units.
• . does not match newline unless DOTALL (re.S, JS s flag).
• Escaping user input inside a pattern: re.escape (Python), Pattern.quote (Java), regexp.QuoteMeta (Go), RegExp.escape in JS runtimes that ship it (Node 24+); otherwise a vetted escape helper. Never interpolate raw input.
• Greedy .* between two delimiters swallows to the last delimiter; use a negated class "[^"]*" rather than lazy ".*?" where possible (clearer and faster).
• Shell quoting: put patterns in single quotes; rg -F or grep -F for literal strings.
• Recompiling in a hot loop costs more than matching; compile once at module scope.
• Translating between engines: lookbehind, possessives, named-group syntax and \Z vs $ differ. Rerun the full test table in the target engine.

Output

**Pattern** (Python `re`, used with `fullmatch`):
`[a-z0-9]+(?:-[a-z0-9]+)*`

**Matches:** `a`, `abc-123`, `a-b-c`  **Rejects:** `-a`, `a--b`, `A`, `a\n`, `ä`
**Linear-time:** yes, each character has one way to match; 100k-char adversarial input in <1 ms
**Notes:** length limit (64) enforced separately before matching