AI Search Optimization skill for Codex

Done well: the bots that power answer engines can fetch the pages, the pages state facts clearly enough to be quoted, the brand is an unambiguous entity across the web, and AI-driven visits are measured. No tricks: answer engines retrieve from search indexes and the open web, so this is mostly excellent SEO plus access decisions.

What is known (2026)

• Google AI Overviews and AI Mode use Google’s Search systems. Google’s guidance: no special markup, files or optimizations are required; the same fundamentals apply (crawlable, indexable, snippet-eligible, helpful content, structured data that matches visible content). Snippet controls (nosnippet, data-nosnippet, max-snippet, noindex) also govern AI features. Traffic from them is included in Search Console’s Web search type with no separate breakdown.
• Google-Extended (robots.txt token) controls use of content for Gemini model training and grounding in some Google products; it does not affect Search or AI Overviews.
• ChatGPT search retrieves through OpenAI’s search crawler (OAI-SearchBot) and third-party search providers; GPTBot is the training crawler; ChatGPT-User fetches pages on a user’s request. Blocking GPTBot does not remove you from ChatGPT search; blocking OAI-SearchBot does.
• Perplexity uses PerplexityBot for its index and Perplexity-User for user-initiated fetches. Microsoft Copilot draws on the Bing index.
• llms.txt is a community proposal (a Markdown file at /llms.txt listing key pages, often with Markdown copies). No major search engine has documented using it for ranking or citation, and Google says AI text files are unnecessary. It can help developer-docs sites whose content is read by coding agents; treat it as cheap and optional, not as a visibility lever.

Vendors rename and add user agents; confirm current names in each vendor’s crawler documentation before writing rules (needs network).

Workflow

- [ ] 1. Decide access policy per bot with the user
- [ ] 2. Verify the bots can actually fetch and read the pages
- [ ] 3. Make content extractable and citable
- [ ] 4. Make the entity unambiguous
- [ ] 5. Optional: llms.txt for docs-heavy sites
- [ ] 6. Measure AI visibility and referrals

1. Access policy

Ask the user what they want; present the trade-off per purpose:

Purpose	Example agents	Block effect
Search index for AI answers	Googlebot, Bingbot, OAI-SearchBot, PerplexityBot	Disappear from those answer engines (and from Google/Bing Search for the first two)
User-initiated fetch	ChatGPT-User, Perplexity-User	Assistants cannot open your page when a user asks
Model training	GPTBot, Google-Extended, CCBot, Applebot-Extended	Content not used for future training; no direct effect on search answers

Common choice for publishers wanting visibility without training use:

User-agent: GPTBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Remember that a named group replaces * rules for that bot; repeat necessary disallows inside each group. Test with sitemaps-robots’s seo_files.py robots.

2. Verify access

• Check CDN/WAF bot protection: many sites block AI crawlers at the edge without knowing (Cloudflare and others offer one-click AI bot blocking). Review those settings with the owner; robots.txt Allow does nothing if the WAF returns 403.
• Server logs: confirm each allowed bot gets 200s. Verify identity via the vendor’s published IP ranges, not just the user-agent string.

grep -E "OAI-SearchBot|ChatGPT-User|GPTBot|PerplexityBot|bingbot|Googlebot" access.log \
  | awk '{print $9, $14}' | sort | uniq -c | sort -rn | head    # adjust fields to log format

• Content must be in server HTML. Many AI fetchers do not execute JavaScript; a client-rendered page may be an empty shell to them even if Googlebot renders it.

curl -s -A "Mozilla/5.0 (compatible; OAI-SearchBot/1.0; +https://openai.com/searchbot)" https://www.example.com/pricing \
  | sed 's/<[^>]*>//g' | grep -c "per month"    # expect the key facts in raw HTML

3. Extractable, citable content

Answer engines quote passages. Make the passage that answers each question stand alone:

• Put a direct, self-contained answer near the top of the section that asks it (“PgBouncer’s transaction mode releases the server connection after each transaction, so …”). Avoid answers that rely on “as mentioned above”.
• State specifics: numbers with units, dates, versions, prices, limits, named comparisons. Vague pages are not cited.
• Publish original data (benchmarks, surveys, pricing indexes) with methodology; it is the content most often cited.
• Keep facts current and dated; stale pricing gets quoted wrongly.
• Use semantic HTML: real headings, lists, <table> for tabular data.
• Comparison and “best X” pages need evidence of testing, not lists.

4. Entity clarity

• One consistent name, description and logo across site, social profiles, directories, app stores, and data sources (Wikidata, Crunchbase, LinkedIn).
• Organization JSON-LD on the homepage with name, url, logo, sameAs links to official profiles; Person pages for authors with credentials. See schema-markup.
• An about page that states plainly what the company does, for whom, where, since when.
• Third-party mentions (reviews, press, community discussions, comparison sites) shape how assistants describe you; inaccurate descriptions usually trace back to an outdated source you can correct.

5. llms.txt (optional)

Worth adding for documentation, API reference or developer tools; skip it for most marketing sites. Format: H1 with the project name, a blockquote summary, then sections of links with one-line descriptions:

# Example API

> Example is a REST API for sending invoices. These links cover authentication,
> core resources and limits.

## Docs
- [Quickstart](https://docs.example.com/quickstart.md): first invoice in 5 minutes
- [Authentication](https://docs.example.com/auth.md): API keys and OAuth scopes
- [Rate limits](https://docs.example.com/limits.md): per-key limits and retry headers

## Optional
- [Changelog](https://docs.example.com/changelog.md)

Generate it from the docs navigation in the build, not by hand, so it does not go stale.

6. Measure

• Analytics referrals from chatgpt.com, perplexity.ai, gemini.google.com, copilot.microsoft.com and similar; ChatGPT often appends utm_source=chatgpt.com. Create a channel group for AI assistants (see ga4-analytics).
• Server logs: fetch counts per AI agent and per URL; user-initiated fetches indicate pages assistants open on behalf of users.
• A fixed prompt panel: 20–50 real questions customers ask, run monthly in each assistant, recording whether the brand is mentioned, cited (linked), and described accurately. Answers vary between runs; track trends, not single answers.
• Search Console cannot isolate AI Overview clicks; watch query-level CTR changes where AI Overviews appear.

Gotchas

• Hidden “instructions for AI” text, cloaked content served only to AI user agents, or prompt-injection strings in pages are manipulative, can be treated as spam, and are easily detected. Never do this.
• Blocking everything “AI” in the WAF also blocks search-retrieval bots and can block Bingbot-based products.
• Do not promise AI citations from schema or llms.txt; no vendor has said either is required for citation.
• Paywalled or login-gated content cannot be cited unless the owner chooses to expose it (Google supports paywalled-content structured data for Search).

Output

**Access:** allowed OAI-SearchBot, PerplexityBot, Bingbot, Googlebot; blocked GPTBot,
Google-Extended, CCBot (owner decision 2026-10-28). WAF AI-bot block disabled for
search agents; logs show 200s for OAI-SearchBot since.
**Rendering:** pricing and docs pages were client-rendered; now SSR (raw HTML has prices).
**Content:** 12 key pages given self-contained answer paragraphs and dated facts.
**Entity:** Organization JSON-LD + sameAs; LinkedIn and Crunchbase descriptions aligned.
**Measurement:** GA4 "AI assistants" channel; monthly 30-prompt panel sheet.