Skill Description Tuning skill for Codex

A tuned description loads the skill for every prompt in its target set, for none of the near-misses, and still works when Codex shortens it because many skills are installed. Treat it like a classifier: write labelled examples first, measure, change one thing, measure again.

How Codex chooses a skill

• At session start Codex lists each available skill’s name, description and file path. That list is budgeted at about 2% of the model’s context window (8,000 characters if the window is unknown); skills.max_context_tokens changes the budget, capped at 10,000 tokens.
• When the list is over budget, Codex shortens descriptions first, then may omit skills entirely and show a warning. The start of each description is what survives.
• Implicit invocation: the model reads that list and decides to open a SKILL.md. Explicit invocation: $skill-name in the prompt or picking from /skills always works, unless the skill is disabled.
• agents/openai.yaml with policy: allow_implicit_invocation: false removes a skill from implicit matching while keeping $name.

Workflow

- [ ] 1. Write the labelled prompt set
- [ ] 2. Measure the current trigger rate
- [ ] 3. Diagnose under- vs over-triggering
- [ ] 4. Rewrite the description (one hypothesis at a time)
- [ ] 5. Re-measure; check siblings did not regress

1. Prompt set

Write 8 to 15 prompts in a file, one per line, prefixed with the expected outcome:

+ write release notes for everything since v2.3
+ what shipped this sprint? draft the changelog entry
+ prepare the GitHub release text for 4.1.0
- summarize the last 5 commits for standup
- bump the version in package.json to 4.1.0
- fix the failing release workflow in .github/workflows

+ must load the skill; - are near-misses that share vocabulary but need something else. Phrase them the way users actually type: short, lowercase, with file names and jargon.

2. Measure

Run each prompt read-only and check whether the skill’s SKILL.md was read:

skill_dir=".agents/skills/release-notes"
while IFS= read -r line; do
  want="${line:0:1}"; prompt="${line:2}"
  codex exec --ephemeral --sandbox read-only --json \
    "$prompt. Do not change any files; stop after deciding your approach." > /tmp/run.jsonl 2>/dev/null
  if grep -q "$skill_dir/SKILL.md" /tmp/run.jsonl; then got="+"; else got="-"; fi
  [ "$want" = "$got" ] && r=ok || r=MISS
  printf '%s want=%s got=%s  %s\n' "$r" "$want" "$got" "$prompt"
done < prompts.txt

This costs one model run per prompt; keep the set small and rerun only after a change. Results vary run to run, so run borderline prompts twice. For a reusable harness, see skill-evals.

3. Diagnose

Symptom	Usual cause	Fix
Never loads implicitly	Description says what the skill IS, not when it applies; abstract nouns	Add “Use when …” with the literal phrases, file names and tools users mention
Loads only when the exact name is used	Trigger words hidden at the end and cut by shortening	Front-load the main use case in the first 100 characters
Loads on unrelated tasks	Broad verbs (“helps with code”, “any documentation”) or a long list of loosely related topics	Narrow the scope; name the artifact and the situation
Loses to a sibling skill	Overlapping vocabulary	Add a boundary: “Not for X; that is covered by <other skill>” in both skills
Two skills both load	Same name in two locations, or near-identical descriptions	Rename one; merge or split responsibilities
Some skills missing entirely	Total descriptions exceed the budget (warning shown)	Shorten all descriptions, disable unused skills, or raise skills.max_context_tokens
Should only run on demand (deploys, destructive ops)	Implicit matching is a risk	allow_implicit_invocation: false in agents/openai.yaml

4. Rewrite

Structure that works:

1. Sentence one, front-loaded: action verb + concrete objects + tools/formats. “Drafts user-facing release notes from merged PRs and conventional commits between two git tags.”
2. Scope detail that also adds keywords: “…grouping by feature, fix and breaking change and linking PR numbers.”
3. Trigger clause: “Use when asked for release notes, a changelog entry or ‘what shipped since vX’, or when preparing a GitHub release.”
4. Optional boundary: “Not for version bumps or CI release workflows.”

Rules:

• Third person, plain text, no markdown or angle brackets, under 1024 characters; aim for 300 to 600.
• Prefer the words users type over internal jargon; include both when they differ (“changelog” and “release notes”).
• File types and tool names are strong signals: Dockerfile, config.toml, EXPLAIN.
• Do not stuff synonyms; each added topic is another chance to over-trigger.
• The name matters too: release-notes beats rn-helper.
• Change one element per iteration so you know what moved the numbers.

Before and after:

# before: never fires implicitly, fires on any "docs" request when it does
description: A helpful skill for documentation and release management tasks.

# after
description: Drafts user-facing release notes from merged PRs and conventional commits between two git tags, grouping features, fixes and breaking changes with PR links. Use when asked for release notes, a changelog entry or "what shipped since v2.3", or when preparing a GitHub release. Not for version bumps or release CI workflows.

5. Re-measure

Rerun the prompt set. Then rerun the prompt sets of sibling skills whose descriptions you touched or whose territory overlaps. Keep the prompt file next to the skill (for example evals/triggers.txt, mentioned in SKILL.md) so the next editor can repeat it.

Gotchas

• Measuring with the skill named in the prompt proves nothing about implicit triggering.
• A session started before the edit keeps the old list; start a new session or run codex exec, and restart Codex if the change does not appear.
• Repo skills apply only when Codex starts inside that repository (any directory up to the repo root); a skill “not triggering” may simply not be in scope.
• Over-triggering is the costlier failure: a wrongly loaded skill spends context and can push Codex into the wrong procedure. When in doubt, narrow.
• Disabled skills ([[skills.config]] with enabled = false) do not load even by name; check before debugging the description.

Output contract

**Skill:** release-notes
**Before:** 3/6 target loaded, 2/5 near-miss loaded
**Change:** front-loaded "release notes"; added trigger phrases; boundary vs version-bump
**After:** 6/6 target, 0/5 near-miss (borderline prompts run twice)
**Siblings rechecked:** changelog-writing 5/5, 0 regressions