Synthetic Monitoring skill for Codex

Done well means: the few journeys that make money or lock users out are exercised every few minutes from outside, a failure means the journey is really broken (not that a selector moved), each run leaves production data clean, and an alert includes the screenshot, trace and step that failed.

Inspect first

1. Read AGENTS.md; look for existing Playwright tests (playwright.config.*, tests/e2e/), a checkly.config.ts, or other synthetic tooling (Datadog Synthetics, Grafana synthetic monitoring with k6 browser).
2. List candidate journeys with the product owner: login, signup, search, add to cart, checkout with a test payment method, core API write and read. Pick three to six.
3. Ask how test accounts and test payments work in production (feature flag, test tenant, payment provider test mode on a separate key). Never use real customer accounts or real payment methods.

Design rules

Rule	Why
One journey per check, short (under 60 s)	A failure points to one flow; long scripts flake
Dedicated synthetic accounts, tagged ([email protected])	Excludable from analytics and billing; safe to reset
Stable selectors: roles, labels, data-testid	CSS classes and text change with design work
Assert outcomes users care about	“Order confirmation number visible”, not “spinner disappeared”
Clean up what the check creates, or use an idempotent path	Production fills with junk orders otherwise
Secrets from the platform’s environment variables	Never committed in the check file
Exclude synthetic traffic from SLIs and product analytics, or tag it	Avoid skewing both

Checkly as code

Initialise with npm create checkly@latest (network required). Layout:

// checkly.config.ts
import { defineConfig } from 'checkly';
import { Frequency } from 'checkly/constructs';

export default defineConfig({
  projectName: 'Shop production monitors',
  logicalId: 'shop-prod-monitors',
  checks: {
    frequency: Frequency.EVERY_5M,
    locations: ['us-east-1', 'eu-west-1', 'ap-southeast-1'],
    tags: ['prod'],
    checkMatch: '**/__checks__/**/*.check.ts',
    browserChecks: { testMatch: '**/__checks__/**/*.spec.ts' },
  },
  cli: { runLocation: 'eu-west-1' },
});

Browser check (plain Playwright Test):

// __checks__/login.spec.ts
import { test, expect } from '@playwright/test';

test('user can log in and see dashboard', async ({ page }) => {
  await page.goto(process.env.APP_URL ?? 'https://app.example.com/login');
  await page.getByLabel('Email').fill(process.env.SYNTH_USER!);
  await page.getByLabel('Password').fill(process.env.SYNTH_PASSWORD!);
  await page.getByRole('button', { name: 'Sign in' }).click();
  await expect(page.getByRole('heading', { name: 'Dashboard' })).toBeVisible({ timeout: 15_000 });
});

API check:

// __checks__/orders-api.check.ts
import { ApiCheck, AssertionBuilder } from 'checkly/constructs';

new ApiCheck('orders-api-read', {
  name: 'Orders API read',
  degradedResponseTime: 1000,
  maxResponseTime: 3000,
  request: {
    method: 'GET',
    url: 'https://api.example.com/v1/orders?limit=1',
    headers: [{ key: 'Authorization', value: 'Bearer {{SYNTH_API_TOKEN}}' }],
    assertions: [
      AssertionBuilder.statusCode().equals(200),
      AssertionBuilder.jsonBody('$.items').isNotNull(),
    ],
  },
});

Commands:

npx checkly test            # runs checks once in Checkly's cloud from runLocation, deploys nothing
npx checkly test --record   # same, with results stored for review
npx checkly deploy          # creates or updates production monitors

npx checkly deploy changes live monitoring and alerting for the team; show the user the diff of checks and get confirmation first, or run it from CI after review. Store SYNTH_USER, SYNTH_PASSWORD and tokens as Checkly environment variables (or secrets), not in the repo.

Self-hosted alternative

When a SaaS is not an option, run Playwright on a schedule and export results:

• Kubernetes CronJob or a CI scheduled workflow per region running npx playwright test --project=synthetic.
• Push synthetic_check_success{journey="login",region="eu"} and duration to Prometheus Pushgateway or via OTLP, and alert with quorum (two regions failing).
• Keep traces and screenshots (trace: 'retain-on-failure', screenshot: 'only-on-failure') in object storage and link them from the alert.

Scheduled CI runners are often not reliable enough to page on (queue delays, 5-minute minimum schedules); use them for tickets, not pages, unless the runner is dedicated.

Alerting

• Retry once in a different location before declaring failure (Checkly retry strategy, or a second run) to remove single-run network flakes.
• Page when the journey fails from two or more locations for two consecutive runs; ticket for a single location or degraded response time.
• Alert payload must include journey name, failing step, screenshot or trace link, and the runbook.
• Route to the team that owns the journey, not to a central platform team.

Flake control workflow

• Find the failing step from traces of the last 20 failures
• Replace brittle selectors with roles, labels or data-testid
• Replace sleeps with web-first assertions (await expect(...).toBeVisible())
• Block third-party scripts that are not part of the journey (page.route for analytics and chat widgets)
• Raise timeouts only for steps with a known slow backend, never globally
• If a flow is inherently slow or nondeterministic, split it or move it to a ticket-level check

Verification

• Run each check locally or with npx checkly test against production and staging.
• Break it on purpose: point at a staging build with the login button renamed, or use an invalid password, and confirm the check fails at the right step with a screenshot.
• Confirm cleanup: after a day of runs, synthetic accounts have no accumulated orders or data beyond what you expect.
• Check the failure rate over the first week; anything above roughly 1% failed runs without a real incident needs flake work before it is allowed to page.

Output

A table of checks: journey, type (browser/API), frequency, locations, assertions, account used, cleanup method, alert route. Include the config files and the deploy command run (or the CI job that runs it).

Gotchas

• Bot protection, CAPTCHAs and WAF rules block synthetic traffic; ask the owner to allowlist the monitor’s IPs or headers rather than trying to bypass them.
• MFA on synthetic accounts needs a supported method (TOTP secret held as a monitor secret) agreed with the security team.
• Checks that create orders can trigger real emails, fulfilment or invoices; use flagged test tenants and confirm the downstream behaviour first.
• Reusing full e2e test suites as monitors makes them slow and flaky; extract only the critical path.