Web Scraping skill for Codex

A good scraper asks for the fewest pages possible, at a pace the site would not notice, identifies itself, survives a crash halfway through without refetching everything, and fails loudly when the page layout changes instead of writing empty rows.

Before writing code

1. Look for a better source. Check for a public API, a JSON endpoint the page itself calls (browser DevTools, Network tab, filter Fetch/XHR), an RSS feed, a sitemap (/sitemap.xml), or a data export. Parsing JSON is ten times sturdier than parsing HTML.
2. Check permission. Read /robots.txt and the site’s terms. Scraping behind a login, collecting personal data, or republishing copyrighted content needs the user’s explicit confirmation that they have the right to do it. Never bypass CAPTCHAs, paywalls or access controls; stop and report instead.
3. Decide whether JavaScript is needed. curl -s URL | grep -c 'expected text'. If the data is in the raw HTML (or in a <script type="application/ld+json"> or __NEXT_DATA__ blob), use plain HTTP. Only if it is rendered client-side, use a browser ($browser-automation or Crawlee’s PlaywrightCrawler).
4. Read AGENTS.md for the project’s Python or Node tooling. Installing packages needs network; in a Codex sandbox with network off, ask for approval first. Every fetch in this skill needs network access too.

Tool choice

Job	Default
A few known URLs, static HTML	httpx or requests + BeautifulSoup (lxml parser)
Crawl many pages, follow links, queue, retries, persistence	Crawlee (BeautifulSoupCrawler in Python, CheerioCrawler in Node)
Client-rendered pages	Crawlee PlaywrightCrawler, or call the underlying JSON API directly
One-off table from one page	pandas.read_html(url) then check it

Politeness defaults

• One request at a time per host, with a delay of 1 to 3 seconds plus jitter. Use the Crawl-delay from robots.txt when it is larger.
• A descriptive User-Agent with contact info (acme-research-bot/1.0 (+mailto:[email protected])). Do not impersonate a browser to evade a block.
• Honour 429 and 503 with Retry-After; back off exponentially; stop the run after repeated 403s instead of rotating IPs or headers.
• Cache raw responses to disk during development so parser iterations do not refetch.
• Scrape off-peak for large jobs and cap the total (max_pages) on every run.

Plain HTTP scraper (Python)

import json, random, time
from pathlib import Path
from urllib import robotparser
from urllib.parse import urljoin

import requests
from bs4 import BeautifulSoup
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry

UA = "acme-catalog-bot/1.0 (+mailto:[email protected])"
BASE = "https://example.com"

rp = robotparser.RobotFileParser(urljoin(BASE, "/robots.txt"))
rp.read()
delay = max(rp.crawl_delay(UA) or 0, 1.5)

session = requests.Session()
session.headers["User-Agent"] = UA
session.mount("https://", HTTPAdapter(max_retries=Retry(
    total=5, backoff_factor=2, status_forcelist=[429, 500, 502, 503, 504],
    allowed_methods=["GET"], respect_retry_after_header=True)))

out = Path("products.jsonl")
done = {json.loads(l)["url"] for l in out.open()} if out.exists() else set()

def parse(html: str, url: str) -> dict:
    soup = BeautifulSoup(html, "lxml")
    title = soup.select_one("h1.product-title")
    price = soup.select_one("[data-testid=price]")
    if not title or not price:
        raise ValueError(f"layout changed? missing fields on {url}")
    return {"url": url, "title": title.get_text(strip=True),
            "price": price.get_text(strip=True)}

def scrape(urls: list[str]) -> None:
    with out.open("a", encoding="utf-8") as f:
        for url in urls:
            if url in done or not rp.can_fetch(UA, url):
                continue
            r = session.get(url, timeout=(5, 30))
            r.raise_for_status()
            f.write(json.dumps(parse(r.text, url), ensure_ascii=False) + "\n")
            f.flush()                                  # checkpoint per record
            time.sleep(delay + random.uniform(0, 1))

Key points: the output file doubles as the checkpoint, the parser raises on missing fields, and timeouts are always set (requests has no default timeout).

Crawlee crawl (Python)

import asyncio
from crawlee.crawlers import BeautifulSoupCrawler, BeautifulSoupCrawlingContext

async def main() -> None:
    crawler = BeautifulSoupCrawler(max_requests_per_crawl=500, max_request_retries=3)

    @crawler.router.default_handler
    async def handle(ctx: BeautifulSoupCrawlingContext) -> None:
        await ctx.enqueue_links(selector="a.pagination-next, a.product-link")
        h1 = ctx.soup.select_one("h1.product-title")
        if h1:
            await ctx.push_data({"url": ctx.request.url, "title": h1.get_text(strip=True)})

    await crawler.run(["https://example.com/catalog"])
    await crawler.export_data("products.json")

asyncio.run(main())

Node equivalent: new CheerioCrawler({ requestHandler: async ({ $, request, enqueueLinks, pushData }) => {...}, maxRequestsPerCrawl: 500, maxConcurrency: 2 }). Crawlee stores its queue and datasets in ./storage; a rerun resumes unless you purge it. Cap concurrency (ConcurrencySettings(max_concurrency=2) in Python) because Crawlee scales up aggressively by default. Crawlee also checks robots.txt only when you enable it (respect_robots_txt_file=True in recent versions); verify for your installed version.

Selector rules

Prefer	Avoid
[data-testid=price], [itemprop=price], semantic ids	div > div:nth-child(3) > span
JSON-LD (script[type="application/ld+json"]) parsed with json.loads	Regex over HTML
Text anchors: find the <dt> labelled “SKU”, take the next <dd>	Class names with hashes (css-1x9k2ab)
Pagination via the “next” link	Guessing ?page=N until a 404

Data hygiene

• Normalise as you go: prices to Decimal plus currency, dates to ISO 8601, whitespace collapsed, relative URLs resolved with urljoin.
• Store the source URL and fetch timestamp on every record.
• Deduplicate by a stable key (canonical URL or SKU), not by the whole row.
• Personal data (names, emails, phone numbers) is out of scope unless the user confirms a lawful basis; never collect more fields than asked.

Gotchas

• response.text guesses encoding from headers; for mojibake, use response.content with BeautifulSoup and let it sniff, or set r.encoding = r.apparent_encoding.
• A 200 page can still be a block page or consent wall. Assert on a content marker, not on the status code.
• Infinite calendars, faceted filters and session IDs in URLs create endless crawls. Restrict enqueue_links with selectors or glob patterns and strip query noise.
• Sites serve different HTML to different locales or currencies; pin Accept-Language.
• Lazy-loaded lists often have a JSON endpoint with ?offset=; use it instead of scrolling.

Verification

• Run against 5 to 10 URLs first and inspect every field by eye against the live page
• Count records with empty or default fields; more than 1 percent means a selector is wrong
• Rerun: already-scraped URLs are skipped and nothing is duplicated
• Kill the run midway and restart: it resumes from the checkpoint
• Request log shows the configured delay and no burst of retries

When a check fails, save the failing page’s HTML to debug/ and fix the selector against that file, not against repeated live fetches.

Report

**Source:** example.com/catalog (robots.txt allows /catalog, crawl-delay 2 s)
**Method:** requests + BeautifulSoup, 1 req / 2-3 s, retries on 429/5xx
**Output:** products.jsonl, 1,284 records (1,290 URLs; 6 returned 404)
**Field quality:** price missing on 3 records (out of stock pages), listed in missing.csv
**Rerun:** `python scrape.py` resumes; delete products.jsonl to start over