Technical Interviews skill for Codex

A good loop predicts on-the-job performance and treats every candidate the same: each competency the role needs is assessed by exactly one or two stages, every stage has pre-written questions and an anchored rubric, interviewers score independently before they talk, and the hire decision follows a rule written before the first candidate.

Workflow

1. Start from the role, not from a question bank. Read the job description and career ladder level; ask the hiring manager for the 4-6 competencies that separate a strong hire from an adequate one at this level.
2. Build the competency matrix: which stage assesses which competency.
3. Choose exercise formats per stage (table below), favouring work samples close to the real job.
4. Write questions, follow-ups, and a 4-level anchored rubric per competency.
5. Write the scorecard, debrief process and decision rule.
6. Pilot with two internal engineers at the target level; fix timing and unclear prompts.
7. Train interviewers and shadow/reverse-shadow before they run a stage alone.

Competency matrix

| Competency | Recruiter screen | Technical screen | Pairing exercise | System design | Behavioural |
|---|---|---|---|---|---|
| Coding fluency in a real codebase | | primary | primary | | |
| Debugging and reading unfamiliar code | | | primary | | |
| System design and trade-offs | | | | primary | |
| Collaboration and communication | | secondary | secondary | secondary | primary |
| Ownership and delivery | | | | | primary |
| Motivation and logistics | primary | | | | |

Rules: every competency has one primary stage; no stage assesses more than three competencies; nothing on the matrix is absent from the job description.

Choosing formats

Format	Signal	Cost	Use when
Pairing on a realistic small task in a prepared repo	High for day-to-day coding	60-90 min	Default for most engineering roles
Debugging exercise (failing test, bug report)	High for maintenance-heavy roles	45-60 min	Roles that inherit existing systems
Code review of a prepared PR	High for senior roles	45 min	Senior and staff levels
System design discussion	High for senior, low for junior	60 min	Mid-level and above
Take-home	High, but excludes candidates with less free time	Candidate’s evening	Only if time-boxed to 2-3 hours, optional alternative offered, or paid
Algorithm puzzles on a whiteboard	Low job relevance for most roles	45 min	Only when the role truly needs algorithm design
Behavioural (past situations)	Good for ownership, collaboration	45 min	Every loop

Total candidate time for the onsite or virtual loop: aim for 4 hours or less.

AI assistance policy

Candidates now have AI coding assistants; decide the policy per stage and tell candidates in advance:

Option	How to run it
AI allowed	Candidate shares screen and uses their normal tools; assess problem decomposition, verification of generated code, and judgment. Make the task large enough that blind acceptance fails.
AI not allowed	Say so in writing; use a live, conversational format where reasoning is discussed aloud
Mixed	Allow in pairing (mirrors the job), disallow in a short fundamentals discussion

Never accuse a candidate of undisclosed AI use based on a hunch; evaluate what you can observe and follow up with questions that require their reasoning.

Writing a stage

Each stage gets a one-page interviewer guide:

## Stage: Pairing exercise (75 min) - competencies: coding fluency, debugging
Setup: repo `interview-orders-api` (Python, 400 lines, tests passing). Candidate gets it
48 hours early with a README; no prep required.
Task: add a discount-code endpoint; existing test suite must stay green.
Timeline: 5 intro | 55 working | 10 extension question | 5 candidate questions
Prompts if stuck (use in order): 1) "What does the test output tell you?" 2) point to
`pricing.py`. Record which prompts were needed.
Extension: "Two requests apply the same single-use code at once. What happens?"
Do not: change the task mid-way, give hints beyond the list, discuss other candidates.

Anchored rubric

Score each competency on 1-4 with behavioural anchors, never on gut feel. Example for “Debugging and reading unfamiliar code”:

Score	Anchor
4 Strong hire	Forms a hypothesis from evidence within minutes, narrows it with targeted experiments, explains root cause and adds a regression test unprompted
3 Hire	Finds the bug with a systematic approach; may need one prompt; fix is correct and tested
2 No hire	Changes code by trial and error; finds the bug only with several prompts; cannot explain why the fix works
1 Strong no hire	Does not find the bug with all prompts; misreads basic program flow

No 2.5s. If evidence is missing for a competency, mark “not assessed” rather than guessing.

Scorecard and debrief

• Interviewers write the scorecard within 24 hours and before seeing anyone else’s: competency scores, evidence (what the candidate said or did, quoted where possible), and an overall recommendation.
• Evidence, not adjectives: “added a test for the empty-cart case before fixing” rather than “seemed thorough”.
• Debrief led by someone without a vote on the outcome (or the hiring manager with a written agenda): go competency by competency, discuss disagreements by evidence, then apply the decision rule.
• Decision rule example: hire if no competency scores 1, at least one 4 in a primary competency, and the average across primary competencies is 3.0 or higher. The hiring manager may override only with a written reason.

Fairness and legality

• Ask every candidate for the role the same core questions; follow-ups may vary.
• Never ask about age, family plans, religion, nationality, health, disability or similar protected characteristics. Ask “Are you authorised to work in <country>?” not where someone is from.
• Offer accommodations proactively (extra time, captions, different format) and make the request route clear in the invitation.
• Track pass-through rates per stage. A stage where pass rates differ sharply between groups, or that rarely changes the outcome, needs review.

Calibration

• Each new question or exercise is piloted on at least two current employees at the target level; their scores set expectations for a 3.
• Interviewers shadow twice, then are reverse-shadowed once before running a stage solo.
• Quarterly: compare interview scores against performance of hires after 6-12 months and retire questions that do not discriminate between strong and weak hires.

Gotchas

• Puzzle-heavy loops select for interview practice, not job performance, and drive away experienced candidates.
• Unbounded take-homes (“should take about a weekend”) are a candidate-experience tax and skew the pipeline toward people with spare time.
• “Bar raiser” or veto rules without written criteria become one person’s taste.
• Letting interviewers chat between stages anchors later interviewers.
• Candidates remember how they were treated. Send the agenda, interviewer names and format in advance, and give a decision within the timeline promised.

Output

# Interview loop - <role, level>
1. Competency matrix
2. Stage guides (one per stage: competencies, setup, task, timeline, prompts, extension)
3. Rubrics with 1-4 anchors per competency
4. Scorecard template and debrief agenda
5. Decision rule and override process
6. Candidate-facing description of the process (include AI policy and accommodations)
7. Pilot results and calibration notes