Technology Evaluation skill for Codex
Pick the library your on-call team will thank you for. Codex fixes constraints and criteria before looking at candidates, eliminates options with pass/fail gates, and scores the short list with evidence for every cell, including release cadence, bus factor, OpenSSF Scorecard and licence history. A bundled script normalises the weighted matrix and shows whether changing any single weight would flip the winner.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- technology-evaluationInstalls to ~/.agents/skills/technology-evaluation/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install technology-evaluationThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project technology-evaluationInstalls to .agents/skills/technology-evaluation/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/technology-evaluation.zipunzip -o technology-evaluation.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) technology-evaluationPowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $technology-evaluation, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Evaluates libraries, frameworks, databases and SaaS vendors with must-have gates, a weighted scoring matrix backed by evidence, maintenance-health checks (release cadence, bus factor, OpenSSF Scorecard, licenses), a time-boxed proof of concept, and a weight-sensitivity check before recommending. Use when choosing between libraries or tools, comparing vendors, asked "should we use X or Y", or writing up a technology choice for review.
Use it when
- You are choosing between two or more libraries, frameworks or services
- A team debate needs an evidence-backed comparison instead of opinions
- You need to check whether a dependency is healthy enough to adopt
Use something else when
- You are running a formal procurement or RFP with contracts: use vendor-evaluation
- The comparison is purely a performance benchmark: use benchmark-comparisons
- The choice is made and you only need to record it: use architecture-decision-records
How to check the result
Every score cites evidence or a PoC number, the script's sensitivity check is reported, and each dropped option has a one-line reason.
The mistake it prevents
Writing the criteria after seeing the options, which quietly tailors the matrix to a favourite chosen in advance.
A good evaluation is decided by the team’s actual constraints, not by feature lists. Hard requirements eliminate options before anything is scored, every score points at evidence (a doc, a measurement, a PoC result), and the recommendation survives reasonable changes to the weights.
Workflow
- [ ] 1. Write the problem, constraints and decision deadline
- [ ] 2. Define gates (pass/fail) and weighted criteria, before looking at options
- [ ] 3. Long list (5-8), cut to a short list of 2-4 with the gates
- [ ] 4. Collect evidence per criterion; run health checks
- [ ] 5. Time-boxed PoC on the riskiest criterion
- [ ] 6. Score, run sensitivity, decide
- [ ] 7. Write the recommendation (and an ADR if the repo uses them)1. Frame the decision
Read AGENTS.md, the dependency manifests and existing architecture docs first: the
current stack, language versions, deployment target and license policy are constraints.
Problem: durable background jobs with retries for the Node API (~2M jobs/day)
Constraints: Node 22, Postgres + Redis already run in prod, AWS, team of 4, no new on-call system
Decision by: 2026-11-15 | Reversibility: medium (job definitions touch ~40 files)Reversibility sets effort: a cheap-to-swap utility library needs an hour; a database or vendor with lock-in deserves a PoC.
2. Criteria before candidates
Writing criteria after seeing options bakes in a favourite. Use gates for anything non-negotiable and weights (summing to 100) for the rest.
| Kind | Typical criteria |
|---|---|
| Gates | Runs on our platform/language; license compatible with our distribution model; data residency; SSO/SOC 2 for vendors; budget ceiling |
| Fit | Solves the core use case; performance on our workload; API ergonomics |
| Operations | Operational burden, observability, upgrade path, failure modes |
| Health | Release cadence, maintainer count, issue response, security posture, funding |
| People | Team familiarity, hiring market, quality of docs |
| Cost | Licence or subscription at 1x and 10x scale, infra, migration effort |
3. Short list
Find candidates from the ecosystem’s own sources (package registry search, awesome lists, the docs of tools you already use), then apply gates. Record why each long-list option was dropped in one line.
4. Evidence and health checks
Collect evidence per criterion with links. Health signals need network (approve if the sandbox blocks it):
npm view bullmq time --json | tail -5 # release dates
npm view bullmq license maintainers
gh api repos/taskforcesh/bullmq --jq '{stars: .stargazers_count, pushed: .pushed_at, open_issues: .open_issues_count, archived: .archived}'
gh api "repos/taskforcesh/bullmq/contributors?per_page=10" --jq '.[] | "\(.contributions)\t\(.login)"'
curl -s https://api.deps.dev/v3/systems/npm/packages/bullmq | jq '.versions[-1]'
scorecard --repo=github.com/taskforcesh/bullmq # OpenSSF Scorecard; needs GITHUB_AUTH_TOKEN| Signal | Healthy | Concern |
|---|---|---|
| Last release | Within 6 months | Over 12 months with open bugs |
| Active maintainers | 3+ with merge rights | One person does over 80% of commits |
| Issue response | Maintainer reply within weeks | Hundreds of unanswered issues |
| Security | Advisories handled, Scorecard 6+ | Unpatched CVEs, no security policy |
| License | Permissive or matches policy | AGPL, SSPL or BSL for a distributed or hosted product (check with legal) |
| Governance | Foundation or several companies | Single vendor that recently relicensed similar projects |
Stars measure popularity, not quality. Weight them low.
For SaaS vendors add: pricing at 10x volume, data export format, SLA and status-page
history, SOC 2 / ISO reports, sub-processors, contract exit terms. Procurement and RFPs
go to $vendor-evaluation.
5. Proof of concept
Spike only the riskiest unknown, time-boxed (half a day to two days), in a branch or scratch directory:
- Use production-shaped data and the real deployment target where possible.
- Define pass criteria before starting (“p95 enqueue < 10 ms at 500 jobs/s on a t3.medium”).
- Record numbers, surprises, and hours spent: effort itself is evidence of ergonomics.
- Follow
$benchmark-comparisonsif the PoC compares performance.
Never commit PoC code to main or add the dependency to the real manifest without the user agreeing to the choice.
6. Score and test sensitivity
Score 1-5 per criterion, each cell with a one-line justification. Then run the bundled script, which drops options failing a gate, normalises totals to 100 and checks whether a +/-50% change to any single weight changes the winner:
python3 scripts/score_matrix.py matrix.csv
python3 scripts/score_matrix.py matrix.csv --swing 0.3criterion,weight,BullMQ,Temporal,Celery
Runs on our stack (Node),gate,1,1,0
Fit for core use case,30,4,5,4
Operational burden,25,4,2,3
Maturity and maintenance,20,4,4,5
Team familiarity,15,5,2,3
Cost,10,5,3,5Interpreting results:
- Lead under 5 points: a tie. Decide on the top risk or on reversibility, and say so.
- Winner flips under one weight swing: the decision hinges on that criterion; confirm its weight with the user explicitly.
- Scores that are all 3s: you lack evidence, not a decision.
Gotchas
- Evaluating the demo, not the day-2 work: upgrades, backups, debugging at 3 a.m.
- Benchmarks from the vendor’s site use their workload. Measure yours.
- Hidden costs: per-seat pricing growth, egress, support tiers, migration effort.
- “Incumbent” bias works both ways: existing tools carry known pain; new tools carry unknown pain. Score both honestly.
- Licence changes (open source to source-available) have happened repeatedly; check the licence history, not just the current file.
- Do not install candidate packages into the project to “try them”. Use a scratch directory or worktree.
Verification
- Every gate and score cell links to evidence or a PoC result.
score_matrix.pyoutput is included, with the sensitivity line.- The recommendation names the main risk and how to detect it early.
- A reviewer can trace why each long-list option was dropped.
Output template
# Evaluation: <capability> (<date>)
**Recommendation:** <option>, with <main risk> mitigated by <action>.
**Decision needed by:** <date> | **Reversibility:** low / medium / high
## Constraints and gates
## Options considered (and why dropped)
## Scoring matrix
| Criterion (weight) | A | B | Evidence |
## Sensitivity
<script output summary>
## PoC results
## Risks and exit planRecord the final choice with $architecture-decision-records if the repo keeps ADRs.
What the zip contains
Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "Technology Evaluation"
short_description: "Pick the library your on-call team will thank you for"
default_prompt: "$technology-evaluation Should we use BullMQ, Temporal or a Postgres queue for background jobs? Score them"scripts/score_matrix.py
#!/usr/bin/env python3
"""Weighted decision matrix with gates and a weight-sensitivity check.
CSV layout (first row is the header; option names follow the weight column):
criterion,weight,optionA,optionB,optionC
License is OSI-approved,gate,1,1,0
Performance on our workload,30,4,3,5
Operational burden,25,3,4,2
- weight "gate": every option needs 1 (pass) or 0 (fail); failing options are excluded.
- numeric weight: any positive number; scores must be 1-5.
Usage: python3 score_matrix.py matrix.csv [--swing 0.5]
Exits 1 on malformed input.
"""
import argparse
import csv
import sys
def fail(msg: str) -> None:
print(f"error: {msg}", file=sys.stderr)
sys.exit(1)
def load(path: str):
with open(path, newline="", encoding="utf-8-sig") as f:
rows = [r for r in csv.reader(f) if r and any(c.strip() for c in r)]
if len(rows) < 2 or len(rows[0]) < 3:
fail("need a header with criterion,weight and at least one option, plus one row")
options = [o.strip() for o in rows[0][2:]]
gates, criteria = [], []
for n, row in enumerate(rows[1:], start=2):
if len(row) != len(rows[0]):
fail(f"line {n}: expected {len(rows[0])} columns, got {len(row)}")
name, weight, cells = row[0].strip(), row[1].strip().lower(), [c.strip() for c in row[2:]]
if weight == "gate":
if any(c not in ("0", "1") for c in cells):
fail(f"line {n}: gate '{name}' scores must be 0 or 1")
gates.append((name, [c == "1" for c in cells]))
continue
try:
w = float(weight)
scores = [float(c) for c in cells]
except ValueError:
fail(f"line {n}: weight and scores must be numbers")
if w <= 0:
fail(f"line {n}: weight must be positive")
if any(s < 1 or s > 5 for s in scores):
fail(f"line {n}: scores must be between 1 and 5")
criteria.append((name, w, scores))
if not criteria:
fail("no weighted criteria found")
return options, gates, criteria
def totals(criteria, alive):
wsum = sum(w for _, w, _ in criteria)
out = {}
for i in alive:
raw = sum(w * s[i] for _, w, s in criteria)
out[i] = 100 * raw / (5 * wsum) # 100 = perfect 5 everywhere
return out
def main() -> None:
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("csv")
ap.add_argument("--swing", type=float, default=0.5, help="relative weight change to test (default 0.5 = +/-50%%)")
args = ap.parse_args()
options, gates, criteria = load(args.csv)
alive = list(range(len(options)))
for name, passes in gates:
for i, ok in enumerate(passes):
if not ok and i in alive:
alive.remove(i)
print(f"GATE FAIL {options[i]}: {name}")
if not alive:
print("No option passes all gates.")
sys.exit(0)
base = totals(criteria, alive)
ranked = sorted(alive, key=lambda i: base[i], reverse=True)
print("\nRank Option Score/100")
for r, i in enumerate(ranked, start=1):
print(f"{r:>4} {options[i]:<22} {base[i]:6.1f}")
if len(ranked) < 2:
return
leader = ranked[0]
margin = base[leader] - base[ranked[1]]
print(f"\nLead of {options[leader]} over {options[ranked[1]]}: {margin:.1f} points")
if margin < 5:
print("Margin under 5 points: treat as a tie and decide on a spike or the top risk.")
flips = []
for k, (name, w, _) in enumerate(criteria):
for factor in (1 - args.swing, 1 + args.swing):
alt = [(n, (w * factor if j == k else ww), s) for j, (n, ww, s) in enumerate(criteria)]
t = totals(alt, alive)
top = max(alive, key=lambda i: t[i])
if top != leader:
flips.append(f" weight of '{name}' x{factor:.2f} -> {options[top]} wins")
print(f"\nSensitivity (each weight x{1 - args.swing:.2f} and x{1 + args.swing:.2f}):")
print("\n".join(flips) if flips else " winner unchanged under every single-weight swing")
if __name__ == "__main__":
main()
About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.