Visual Regression Testing skill for Codex

A visual suite earns its keep when every diff it reports is a real visual change, baselines are rendered in the same environment that compares them, and updating a baseline is a reviewed decision rather than a reflex to turn CI green.

Inspect first

1. Read AGENTS.md, then look for existing visual tooling: toHaveScreenshot in Playwright specs, __screenshots__ or *-snapshots/ folders, a Chromatic project token in CI, Storybook config, or Vitest browser toMatchScreenshot.
2. Find where baselines are generated today. Baselines produced on a developer’s Mac and compared on Linux CI differ in font rendering and will never pass reliably.
3. Check whether Storybook exists. Component stories are the cheapest visual surface.
4. Note that Docker images, browser downloads and Chromatic uploads need network access; Chromatic also needs a project token stored as a CI secret.

Choose the approach

Situation	Approach
Design system or component library with Storybook	Chromatic (hosted review UI, TurboSnap), or Storybook test runner + Playwright screenshots self-hosted
App pages and flows already tested with Playwright	expect(page).toHaveScreenshot() in a dedicated visual project
Components tested with Vitest browser mode	toMatchScreenshot in a browser project
Budget for no hosted service, strict data residency	Self-hosted Playwright baselines in git

Start with a small set: core layout pages and the 10-20 most reused components in their key states. Screenshot everything and the team stops reading diffs.

Deterministic rendering checklist

• Fonts: self-hosted and loaded before capture (await page.evaluate(() => document.fonts.ready)); no network font that may time out
• Animations and transitions disabled (animations: "disabled", plus caret: "hide")
• Time frozen (page.clock.setFixedTime(...)) for relative dates and countdowns
• Data seeded or mocked; no live feeds, ads, avatars from third-party URLs
• Viewport and device scale factor fixed per project
• Dynamic regions masked (timestamps, user ids, charts with random data, maps)
• Lazy images scrolled into view or loaded eagerly in the test build
• Same OS, browser build and fonts for baseline creation and comparison

Playwright setup

// playwright.config.ts (excerpt)
export default defineConfig({
  expect: {
    toHaveScreenshot: { maxDiffPixelRatio: 0.002, animations: "disabled", caret: "hide" },
  },
  snapshotPathTemplate: "{testDir}/__screenshots__/{projectName}/{testFilePath}/{arg}{ext}",
  projects: [
    {
      name: "visual-chromium",
      testMatch: /.*\.visual\.ts/,
      use: { ...devices["Desktop Chrome"], viewport: { width: 1280, height: 800 } },
    },
  ],
});

// e2e/pricing.visual.ts
import { test, expect } from "@playwright/test";

test("pricing page", async ({ page }) => {
  await page.clock.setFixedTime(new Date("2026-03-01T10:00:00Z"));
  await page.route("**/api/plans", (r) => r.fulfill({ path: "e2e/fixtures/plans.json" }));
  await page.goto("/pricing");
  await page.evaluate(() => document.fonts.ready);

  await expect(page).toHaveScreenshot("pricing.png", {
    fullPage: true,
    mask: [page.getByTestId("live-chat-badge")],
  });
});

test("plan card states", async ({ page }) => {
  await page.goto("/pricing");
  const card = page.getByRole("article", { name: "Team plan" });
  await expect(card).toHaveScreenshot("team-card.png");
  await card.hover();
  await expect(card).toHaveScreenshot("team-card-hover.png");
});

• Element screenshots (locator.toHaveScreenshot) are more stable and produce clearer diffs than full pages; use full pages only for layout.
• stylePath can inject a CSS file during capture to hide noise globally (scrollbars, blinking cursors) without changing the app.
• The default template includes platform in file names; the custom template above drops it, so baselines must always be generated in the same container.

Baselines in the CI environment

Generate and compare inside the official Playwright image matching the installed @playwright/test version:

docker run --rm --ipc=host -v "$PWD":/work -w /work \
  mcr.microsoft.com/playwright:v<version>-noble \
  npx playwright test --project=visual-chromium --update-snapshots

--update-snapshots (-u) accepts modes in recent versions (changed, missing, all); missing writes only new baselines without touching existing ones.

Chromatic with Storybook

npx chromatic --project-token="$CHROMATIC_PROJECT_TOKEN" --only-changed --exit-zero-on-changes

• --only-changed (TurboSnap) snapshots only stories affected by the git diff; it needs full git history in CI (fetch-depth: 0).
• --exit-zero-on-changes keeps the build green while visual changes wait for review in Chromatic; the PR status check from Chromatic is the gate.
• --auto-accept-changes main accepts on the main branch after merge so main becomes the new baseline.
• Story-level control through parameters:

export const Loading: Story = {
  parameters: {
    chromatic: { delay: 300, diffThreshold: 0.2, pauseAnimationAtEnd: true },
  },
};
export const Playground: Story = { parameters: { chromatic: { disableSnapshot: true } } };

• Capture each meaningful state as its own story (default, hover via play function, error, empty, long text, RTL, dark mode) instead of one story with knobs.
• Use modes (viewports, themes) only for components that change across them; every mode multiplies snapshot count and cost.

Thresholds

Setting	Use
maxDiffPixels: 0 / ratio 0	static icons, logos in a pinned environment
maxDiffPixelRatio around 0.001-0.005	pages and components, absorbs anti-aliasing
threshold (per-pixel color distance, 0-1)	raise slightly only for gradient or image-heavy areas
Masks	anything whose content legitimately varies

Raising thresholds to silence a flaky diff hides real one-pixel regressions such as a border disappearing. Remove the nondeterminism first.

Review workflow

1. CI fails (or Chromatic marks changes) on a PR.
2. Open the diff artifact or Chromatic review. Classify each diff: intended, regression, or noise.
3. Intended: update baselines in the CI container (or accept in Chromatic) in the same PR, and mention it in the PR description.
4. Regression: fix the CSS; do not update.
5. Noise: fix the source of nondeterminism (font, time, data, animation), then rerun.

Never run --update-snapshots across the whole suite to make a red build green without looking at each diff. Ask the user before bulk-accepting.

Gotchas

• Baselines committed from macOS, compared on Linux: always fails. Regenerate in Docker.
• Git LFS or large PNG churn bloats the repo; keep the set small or use a hosted service.
• Web fonts with font-display: swap capture the fallback font if the test does not wait.
• Hover and focus states leak between screenshots when the mouse stays put; move the mouse away (page.mouse.move(0, 0)) before the next capture.
• Visual tests do not prove accessibility or behavior; keep functional assertions in regular E2E ($playwright-e2e) and component tests.

Verification

1. Run the visual project twice in the CI container without code changes: zero diffs.
2. Change one CSS value (padding, color) deliberately: the affected snapshots fail and the diff image highlights only that area. Revert.
3. Confirm baselines in git were produced by the container, not a local OS.
4. In Chromatic, confirm the PR check is required and TurboSnap reports affected stories.