Survey Design skill for Codex

A survey is done well when every question maps to a decision, each question measures one thing in neutral words, the people answering resemble the people you want to generalise to, and the analysis plan exists before the first response arrives.

Workflow

- [ ] 1. Write objectives and the decisions they inform
- [ ] 2. Decide whether a survey is the right method
- [ ] 3. Define population, sampling frame and target sample size
- [ ] 4. Draft questions mapped to objectives
- [ ] 5. Order, logic, randomisation, length check
- [ ] 6. Pilot (cognitive interviews + soft launch)
- [ ] 7. Write the analysis plan, then field

1-2. Objectives and method fit

Decision: whether to build a self-hosted edition next quarter
Objective O1: share of active accounts that cannot use cloud hosting, and why
Objective O2: what they use instead today
Objective O3: willingness to pay relative to the current plan

A survey measures how common something is. It does not discover why people behave as they do or what they would want that does not exist yet. If you cannot write answer options because you do not know the space, run interviews first ($customer-interviews).

3. Population, frame, sample

Term	Ask
Population	Who do conclusions apply to? (“accounts active in the last 90 days”)
Sampling frame	The list you can actually reach. Who is missing from it?
Sampling method	Random from the frame is the default; stratify when small segments matter
Sample size	From the precision you need (below), adjusted for expected response rate

Sample size for a proportion at 95% confidence:

n0 = 1.96^2 x p(1-p) / e^2        (use p = 0.5 if unknown; e = margin of error)
e = 5%  -> n0 = 385     e = 3% -> n0 = 1,068     e = 10% -> n0 = 97
Finite population: n = n0 / (1 + (n0 - 1) / N)
Invites needed = n / expected response rate (often 5-30% for customer email surveys)

Margin of error applies to each subgroup separately: 400 responses split across four segments give roughly +/-10% per segment. Plan the sample around the smallest segment you need to report.

Margins assume a probability sample. Opt-in, social-media or in-app convenience samples have unknown bias that more responses do not fix; report them as “among respondents”.

4. Writing questions

Rules, each a frequent cause of unusable data:

Problem	Bad	Better
Double-barreled	“How fast and reliable is the API?”	Two questions
Leading	“How much do you love the new dashboard?”	“How would you rate the new dashboard?”
Loaded assumption	“Why did you stop using exports?”	Ask whether they use exports first, then branch
Vague time / quantity	“Do you deploy often?”	“In the last 30 days, how many times did your team deploy to production?”
Jargon	“Rate our RBAC granularity”	“How well can you control who can see and change each project?”
Hypothetical	“Would you pay for X?”	Ask about current spend and behaviour; use pricing methods for WTP
Unbalanced scale	Poor / Good / Very good / Excellent	Very poor … Very good, symmetric around a midpoint
Missing options	No “None”, “Other”, or “Not applicable”	Add them; make “Other (please specify)” text
Overlapping ranges	1-10, 10-50	1-9, 10-49, 50+

Scales:

• Use 5-point labelled scales by default; label every point, not only the ends.
• Keep direction consistent across the survey (negative to positive).
• Agree/disagree batteries invite acquiescence; prefer item-specific scales (“How easy or difficult was …”).
• Do not change a tracking question’s wording or scale between waves; trend data breaks.

NPS uses a 0-10 likelihood-to-recommend item; promoters 9-10, detractors 0-6. Report the distribution and the sample, not only the single score.

5. Order, logic, length

• Start with easy, relevant, non-sensitive questions; demographics and firmographics last unless needed for screening or branching.
• General before specific to avoid priming (overall satisfaction before feature ratings).
• Randomise option order for unordered lists; keep ordered scales fixed.
• Use skip logic so nobody answers about features they do not use.
• Target under 10 minutes (roughly 15-25 closed questions); completion drops with length.
• Limit open text to 1-3 questions; they are expensive to analyse.
• Add one attention check only for paid panels; do not insult customers with them.

6. Pilot

1. Cognitive interviews with 3-5 people from the population: they think aloud while answering. Fix every question someone interprets differently than intended.
2. Soft launch to ~5-10% of the sample. Check completion time, drop-off per page, straight-lining, and “Other” responses that reveal missing options.

7. Analysis plan, ethics, fielding

• Before fielding, write which question answers which objective, the breakdowns, and what result would change the decision.
• State purpose, time required, how data is used and stored, and whether responses are anonymous. Collect only what you will analyse. Personal data falls under GDPR/CCPA; route consent wording to the people who own privacy.
• Sending the survey to customers is outward-facing: get explicit confirmation of the audience list, message and send time before anything is sent.
• Incentives raise response rates but can attract low-effort answers; keep them modest.

Gotchas

• Nonresponse bias: the angriest and happiest customers answer. Compare respondents to the frame on known attributes (plan, tenure, size) and weight or caveat.
• Social desirability: people overreport good practices (testing, security). Ask about specific recent behaviour instead of habits.
• Asking what people would do predicts actual behaviour poorly; use for direction only.
• Required questions on sensitive items raise drop-off and fake answers; make them optional.
• Changing the survey mid-field splits the data into two incompatible sets.

Verification

• Every question maps to an objective; delete those that do not.
• Run the rules table over every item; fix all double-barreled and leading wording.
• Sample size meets the precision target for the smallest reported segment.
• Pilot shows median completion time within target and no misread questions.

Output template

# Survey: <name>
**Decision it informs:** ... | **Population:** ... | **Frame:** ...
**Target completes:** n (margin +/-x% at 95%) | **Invites:** N at y% expected response
**Length:** ~z minutes, q questions

| # | Question | Type / scale | Options | Objective | Logic |
|---|---|---|---|---|---|

## Analysis plan
## Consent text
## Pilot findings and changes