Vendor Evaluation skill for Codex
Vendor picks you can defend a year later. Codex runs vendor selection like a disciplined procurement lead: it sizes the process to the risk, fixes must-haves and weights before any demo, writes the RFP and a scripted demo using your scenarios, checks SOC 2 Type II scope, DPAs and sub-processors, and compares three-year total cost rather than first-year quotes. Scoring is independent and evidence-based, and the result is a recommendation memo with risks and negotiation targets.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- vendor-evaluationInstalls to ~/.agents/skills/vendor-evaluation/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install vendor-evaluationThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project vendor-evaluationInstalls to .agents/skills/vendor-evaluation/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/vendor-evaluation.zipunzip -o vendor-evaluation.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) vendor-evaluationPowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $vendor-evaluation, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Runs vendor selection for software and services - requirements with must-haves and weights fixed before any demo, RFI and RFP documents, a weighted scoring matrix, security and privacy due diligence (SOC 2 Type II, ISO 27001, questionnaires, DPA, sub-processors), three-year total cost of ownership, structured reference calls, proof-of-concept success criteria and a written recommendation. Use when asked to choose a vendor, write an RFP or RFI, compare vendor proposals, score demos, assess vendor risk, or justify a purchase decision.
Use it when
- Buying software or services that are costly, sensitive or hard to replace
- Writing an RFI or RFP and scoring the responses
- Security or legal asks for vendor due diligence evidence
- Leadership wants a documented justification for a purchase
Use something else when
- You are comparing open-source libraries or frameworks for a codebase: use technology-evaluation
- You already picked the vendor and need the contract reviewed: use contract-review-checklist
- You are the vendor writing a proposal: use sales-proposals
How to check the result
Weights were approved before demos, every shortlisted vendor has scores with justifications on the same criteria, security evidence is dated and in scope, and TCO covers three years including exit cost.
The mistake it prevents
Scoring roadmap promises as current capability after a polished vendor-led demo instead of a scripted one using your own data.
A sound vendor decision can be defended a year later: requirements and weights were fixed before anyone saw a demo, every vendor got the same questions and the same evaluation, security and contract risk were checked with evidence rather than marketing pages, and the cost compared is three-year total cost, not the first-year quote.
Size the process to the risk
| Tier | Example | Process |
|---|---|---|
| Low | Under ~$10k/year, no sensitive data, easy to replace | Short requirements list, 2-3 options, trial, owner decides |
| Medium | Team-wide tool, customer data, moderate integration | Requirements + scoring, demos, security review, references |
| High | Core system, regulated or large volumes of personal data, multi-year, hard to exit | RFI to shortlist, formal RFP, POC, full security and legal review, executive approval |
Ask the user for budget, data sensitivity and switching cost, then choose the tier.
Workflow
- Problem statement and success measures (what changes when the vendor is live)
- Stakeholders: decision owner, users, IT/security, legal, finance, procurement
- Requirements list with must-have vs weighted criteria, signed off before outreach
- Long list from market scan (needs network; otherwise from the user’s list)
- RFI or quick screen against must-haves to reach a shortlist of 3-4
- RFP or structured demo script for the shortlist
- Security, privacy and compliance due diligence
- TCO comparison
- Reference calls and, for high tier, POC against written success criteria
- Score, recommend, negotiate, record the decision
Requirements and weights
Must-haves are pass/fail; failing one eliminates the vendor regardless of score. Weighted criteria distinguish the survivors.
| ID | Requirement | Type | Weight | How verified |
|---|---|---|---|---|
| M1 | SSO via SAML or OIDC with SCIM provisioning | must | - | demo + docs |
| M2 | Data hosted in EU region | must | - | contract + DPA |
| W1 | Workflow automation without code | weighted | 20% | demo script task 3 |
| W2 | API coverage for our 4 integration points | weighted | 20% | API docs + POC |
| W3 | Admin and reporting | weighted | 10% | demo script task 5 |
| W4 | Implementation effort and vendor support | weighted | 15% | references + plan |
| W5 | Security posture | weighted | 15% | due diligence |
| W6 | 3-year TCO | weighted | 20% | TCO model |Weights sum to 100% and are agreed by stakeholders in writing before demos. Changing weights after seeing results is how evaluations get rigged, intentionally or not.
RFP structure
- Background and objectives (what you are buying and why)
- Timeline: questions deadline, response deadline, demos, decision date
- Instructions: format, single point of contact, how questions are answered (shared with all bidders), confidentiality
- Requirements: numbered, each answered with Yes / Partial / Roadmap (date) / No plus explanation; ask for evidence links
- Demo script: the scenarios every vendor must show using your sample data
- Security and privacy questionnaire (or a standard one such as SIG Lite or CAIQ)
- Pricing template: identical structure for all vendors (licences, usage, implementation, support tiers, training, renewal caps, payment terms)
- References requested: similar size and use case, at least one former customer if possible
- Evaluation approach (criteria categories, not exact weights)
Sending an RFP or contacting vendors is outward-facing: draft everything, and send only after the user confirms recipients and content.
Security and privacy due diligence
| Evidence | What to check |
|---|---|
| SOC 2 Type II report | Period covered (recent 12 months), scope includes the product you buy, exceptions noted, complementary user entity controls you must implement |
| ISO 27001 certificate | Issuing body, scope statement, validity dates |
| Pen test summary | Date within 12 months, critical findings remediated |
| DPA and sub-processor list | Roles, transfer mechanisms, breach notice timing, sub-processor change notice |
| Data handling | Encryption at rest and in transit, data residency, retention and deletion on exit, backup restore testing |
| Access | SSO, MFA, RBAC, audit logs available to you |
| AI features | Whether your data trains models, opt-out, model providers used |
| Business continuity | RTO/RPO, status page history, incident communication |
A SOC 2 Type I or a “SOC 2 in progress” letter is weaker evidence than a Type II report; note it as a risk rather than a pass. For contract terms, hand off to the contract-review-checklist skill.
Total cost of ownership (3 years)
| Cost | Vendor A Y1 | Y2 | Y3 | Vendor B Y1 | Y2 | Y3 |
|---|---|---|---|---|---|---|
| Licences / subscription (with expected growth in seats/usage) | | | | | | |
| Usage or overage fees | | | | | | |
| Implementation (vendor + partner) | | | | | | |
| Internal staff time (hours x loaded rate) | | | | | | |
| Integrations and middleware | | | | | | |
| Training and change management | | | | | | |
| Support tier | | | | | | |
| Exit / migration cost (estimate) | | | | | | |
| Total | | | | | | |Ask each vendor for renewal price caps; an uncapped renewal makes Y2-Y3 a guess.
Reference calls
Prepare the same questions for every reference; ask the vendor for customers like you, and find one independently if you can (network permitting).
- What problem did you buy it for, and was it solved?
- How long did implementation really take versus the plan?
- What surprised you, good and bad, in year one?
- How does support respond to a severity-1 issue?
- What happened at renewal (price, terms)?
- Would you choose them again? What would you do differently?
Proof of concept
Write success criteria before the POC starts, with pass thresholds and test data: “Sync 50,000 records from our CRM sandbox nightly with zero errors for 5 consecutive days.” Time-box it (2-4 weeks), use your data and your users, and score against the criteria, not against how the vendor’s team felt about it.
Scoring and recommendation
- Each evaluator scores independently on a 1-5 scale per criterion with a one-line justification, then the group reconciles differences above 1 point by discussing evidence.
- Weighted score = sum of (score x weight). Show the sensitivity: would the winner change if the top weight moved by 10 points? If yes, say so.
- Recommendation memo: problem, options, scoring table, TCO, risks and mitigations, negotiation targets, decision needed and by when. See decision-memos for the format.
Gotchas
- Demo theatre: vendors show their best path. Script the demo with your scenarios and data.
- Roadmap items scored as present capability. Score only what exists; record roadmap promises in the contract if they matter.
- Evaluating price before must-haves; the cheapest vendor that fails SSO is not cheap.
- Ignoring exit: data export format, termination assistance and API completeness decide how locked in you are.
- Single-evaluator scoring, or the champion of one vendor also running the process.
- Telling the losing vendors nothing; a short, courteous notice protects future relationships (send only with the user’s approval).
Output
Requirements matrix, RFP or demo script, completed scoring matrix with justifications, security review summary with evidence links, 3-year TCO table, reference call notes, and the recommendation memo with open risks and negotiation points.
What the zip contains
SKILL.md plus the interface file below. Nothing runs on install.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "Vendor Evaluation"
short_description: "Vendor picks you can defend a year later"
default_prompt: "$vendor-evaluation Help us choose a CRM: write requirements with weights and an RFP for four shortlisted vendors"About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.