Beta Programs skill for Codex

A beta is run well when it answers specific questions that internal testing cannot, participants are chosen to represent the launch audience, every piece of feedback lands somewhere it will be read and answered, and general availability is decided against exit criteria written before the beta started.

Define the purpose first

Beta goal: <what we must learn or prove before GA>
Questions:
1. Does the sync engine hold up with real data sizes (10k+ records)?
2. Do new admins complete setup without a call?
Exit criteria (all required for GA):
- Crash-free sessions >= 99.5% over the final 2 weeks
- No open P0/P1 bugs; P2s triaged with owners
- >= 70% of active participants complete setup unaided
- >= 60% of participants say they would be disappointed to lose it (or other agreed bar)
Stop / rollback criteria: data loss of any kind; P0 unresolved for 72 h
Time box: 6 weeks, review weekly

Without exit criteria, betas drift for months and become an unsupported product line.

Choose the beta type

Type	Use when	Size
Internal dogfood	Before any external exposure; obvious bugs remain	Whole company
Closed beta	Need deep feedback, product still changing, risk to data or trust	10-100 accounts
Open beta	Need scale, performance and edge-case coverage; product mostly stable	Anyone who opts in
Early access (paid or waitlist)	Testing willingness to pay or demand	Varies

Default sequence: dogfood, closed, optionally open, then GA.

Participants

• Select against the launch audience: mix of segment, size, plan, platform, region and data volume. Avoid filling the beta with only your friendliest customers.
• Include a few demanding or high-volume accounts; they find the scale bugs.
• Screen for availability: participants who agree to a kickoff call, at least weekly use, and one feedback session.
• Plan for about half of invited participants to be active after week two; invite accordingly.
• Exclude accounts where a failure would be unacceptable (regulated data, launch-week dependencies) unless the beta is specifically for them and they accept the risk.

Mechanics

• Access control: gate with a feature flag per account or user so access can be granted and revoked without deploys ($feature-flags). Name it for the feature, e.g. bulk_import_beta.
• Terms: beta terms or an addendum covering no SLA, possible data changes, confidentiality if needed, and feedback usage. Legal should review the wording.
• Data safety: decide what happens to data created in the beta at GA or rollback; migration paths must exist before anyone puts real work in.
• Labelling: a visible “Beta” badge and a feedback link inside the feature.
• Telemetry: instrument key events before launch ($analytics-tracking-plan); you cannot answer beta questions retroactively.

Feedback channels

Channel	Captures	Owner reads
In-product feedback widget	Moment-of-use issues with context (page, account, version)	Daily
Shared channel (Slack Connect, community forum)	Discussion, quick questions	Daily
Bug form or tracker label beta:<feature>	Reproducible defects	Daily triage
Scheduled interviews / calls	Depth on value and workflow	Weekly
End-of-beta survey	Comparable scores across participants	Once

Every report gets an acknowledgement within one business day and a resolution note when fixed. Participants who never hear back stop reporting. Route incoming feedback through $feedback-triage.

Weekly health review

## Beta week 3 - <feature>
Participants: 42 invited, 31 activated, 22 active this week
Exit criteria status:
| Criterion | Target | Now | Trend |
Bugs: P0 0 | P1 2 (owners) | P2 9
Top themes from feedback (count of accounts):
Decisions this week:
Risks:

Communications

Draft each message and get the user’s approval before it is sent; these go to customers.

• Invitation: what it is, what we ask of them, time commitment, beta terms link, how to opt out.
• Kickoff/onboarding: access steps, feedback channels, known limitations.
• Weekly or biweekly update: what changed thanks to their feedback (name fixes).
• Graduation: GA date, any pricing changes, what happens to their data and settings.
• Wind-down (if stopped): why, timeline, data export path, thanks.

Graduation decision

- [ ] Every exit criterion met, or the owner explicitly accepts a documented gap
- [ ] Support team trained; help docs and release notes ready
- [ ] Pricing and packaging decided for GA
- [ ] Flag rollout plan for GA (percentage ramp) and rollback path
- [ ] Beta participants informed before the public announcement
- [ ] Beta-only scaffolding (badges, special channels) scheduled for removal

If criteria are not met at the time box, choose explicitly: extend with a new end date and a reason, change scope, or stop. Shutting a beta down and removing participant access or data requires confirmation from the user and notice to participants.

Gotchas

• A beta that is actually a soft launch (no questions, no exit criteria) gives a false sense of validation.
• Participants are not testers: do not expect structured bug reports without a form and prompts.
• Beta accounts often get white-glove support that GA customers will not; measure the unaided experience separately.
• Giving permanent free access as a beta reward complicates pricing later; offer a time-limited discount instead, decided up front.
• Survey-based exit bars need enough respondents to mean anything; with fewer than ~20 responses, report counts, not percentages.

Verification

Check that the beta plan has questions, measurable exit and stop criteria and a time box before any invites go out; the flag can revoke access per account; every feedback channel has a named owner and response time; and the weekly review reports each exit criterion with its current value.