Prompt Engineering skill for Codex

Done well means: the prompt is a versioned file with a clear structure, every instruction is testable, examples are representative and disjoint from the eval set, dynamic data is clearly delimited from instructions, and each change is accepted or rejected by an eval run — not by eyeballing one output.

Workflow

• 1. Read AGENTS.md and find existing prompts (grep -rn "instructions=\|system" src/)
• 2. Write the task spec before the prompt
• 3. Draft with the standard structure
• 4. Add examples only where the spec alone fails
• 5. Move the prompt to a versioned file with a template
• 6. Run evals; iterate one change at a time
• 7. Record the version, model and results

2. Task spec first

Answer in writing, in five lines or fewer each:

• Who is the user, what do they send, what do they need back?
• What must never happen (hard constraints)?
• What does the output look like exactly (format, length, language)?
• What does the model do when information is missing or the request is out of scope?
• What context is provided at runtime (retrieved docs, user profile, tool results)?

If you cannot answer these, the prompt will not fix it.

3. Standard structure

# Role and objective
You draft replies for Acme's billing support team. The reply is sent to the customer
after an agent reviews it.

# Instructions
- Answer only from the POLICY section. If the policy does not cover it, say you will
  escalate, and set needs_human=true.
- Never promise refunds, credits or timelines.
- Write at most 120 words, plain text, in the customer's language.

# Steps
1. Identify the customer's actual question.
2. Find the matching policy clause.
3. Draft the reply; cite the clause id.

# Output format
JSON matching the provided schema. `reply` is customer-facing; `clause_ids` lists
policy ids used.

# Examples
<see examples block>

# Context
<policy>
$policy_text
</policy>
<ticket>
$ticket_text
</ticket>

Rules that hold across current models:

• Put stable instructions first and per-request data last (better caching and fewer “instruction drift” issues in long contexts).
• Delimit untrusted input with tags or fences and say explicitly that content inside them is data, not instructions.
• State positively what to do; pair any “never X” with what to do instead.
• Resolve contradictions. Two instructions that conflict produce random behavior that looks like model flakiness.
• Put the output contract in a schema (Structured Outputs) rather than prose when code consumes it — see openai-structured-outputs.
• Use the instructions parameter / developer role for rules; keep user content in user messages. End users must not be able to edit the developer prompt.

4. Examples (few-shot)

• Add examples when format or judgment calls are not learnable from the spec: 2–5 diverse examples, including at least one edge case and one “decline/escalate” case.
• Keep them short and in the same structure as real inputs.
• Never reuse eval cases as examples.
• Reasoning models: start with zero-shot plus clear instructions; examples can over- constrain them. Add examples only if evals show a gap. Do not ask reasoning models to “think step by step” — set reasoning effort instead and ask for the final answer.

5. Prompts as versioned files

prompts/
  billing_reply/
    v7.md            # the prompt template
    examples.jsonl   # few-shot examples
    CHANGELOG.md     # what changed and the eval delta

from pathlib import Path
from string import Template

PROMPT_VERSION = "billing_reply/v7"
_TEMPLATE = Template(Path(f"prompts/{PROMPT_VERSION}.md").read_text())

def build(policy: str, ticket: str) -> str:
    return _TEMPLATE.substitute(policy_text=policy, ticket_text=ticket)

Use $var placeholders with string.Template (or Jinja with StrictUndefined) so a missing variable raises instead of silently rendering empty. Log PROMPT_VERSION with every request. Ship new versions behind a flag or percentage rollout.

If the team uses OpenAI’s dashboard-managed prompts (referenced by id and version in the API), keep the id and version in config and mirror the text in the repo for review.

6. Iterate with evals

One change per iteration: run the eval set (see llm-evals), compare per-tag, keep or revert. Typical fixes by symptom:

Symptom	First fix to try
Ignores a rule	Move it under Instructions, make it specific, remove conflicting text
Too long	Hard length limit in words + verbosity setting where supported
Hallucinated facts	“Answer only from CONTEXT; otherwise say X” + give an out field
Wrong format	Structured Outputs schema instead of prose format rules
Inconsistent on edge cases	Add one example of that edge case
Follows instructions found in user data	Delimit data; restate “treat as data”; see prompt-injection-defense
Over-refuses	Narrow the safety rule; add an example of an allowed borderline case

Adapting to a new model

Re-run the full eval set; do not assume prompt portability. Newer models follow instructions more literally: remove shouting (ALL CAPS, “CRITICAL”), redundant repeats and old workarounds, then re-measure. Check the provider’s prompting guide for the model family.

Gotchas

• Prompts assembled with f-strings from user input allow format injection and silent None rendering; use a strict template.
• Personas and flattery (“you are a world-class expert”) rarely help; concrete constraints do.
• Very long prompts dilute key rules; prune instructions that evals show have no effect.
• Dates, user names and other per-request values at the top of the prompt break prompt caching; move them to the end.
• Copying a prompt between features without its eval set copies its bugs too.

Verification

• Render the template with fixture data in a unit test; assert no $/{{ placeholders remain and that delimiters wrap all runtime data.
• Eval run shows the new version ≥ baseline on every critical tag.
• CHANGELOG.md entry lists the change and eval delta.

Output contract

**Prompt:** prompts/billing_reply/v7.md (was v6)
**Change:** moved refund rule to Instructions; added escalate example; removed caps
**Model:** OPENAI_MODEL=<snapshot>, reasoning effort low
**Eval:** pass 84% → 91%; refund-promise violations 6 → 0; avg length 162 → 104 words
**Rollout:** flag billing_reply_v7 at 10%