RSS and Page Change Monitoring skill for Codex

A monitor is right when it alerts exactly once per genuinely new item, stays silent through timestamp churn and ad rotation, costs the publisher almost nothing per check (304 Not Modified most of the time), and tells you when it has itself stopped working.

Choose the source

Signal you want	Watch
New posts, releases, episodes	The feed (<link rel="alternate" type="application/rss+xml"> in the page head)
GitHub releases or tags	https://github.com/OWNER/REPO/releases.atom or /tags.atom
A value on a page (price, status, stock)	That element via CSS selector, not the whole page
A list without a feed (jobs, changelogs)	The list container; diff the set of item links
An API that exposes updated_since	Poll the API instead ($api-polling-workers)

Ask the publisher’s terms and robots.txt before polling a non-feed page often. Feeds exist to be polled; pages generally are not.

Polling budget

• Feeds: every 15 to 60 minutes is normal. Honour <ttl> (RSS, minutes), Cache-Control: max-age, and Retry-After. Never under 5 minutes without permission.
• Pages: hourly or daily. Back off exponentially after errors; stop after a run of 403/410.
• Always send conditional requests; a well-behaved monitor mostly receives 304s.
• Fetching needs network; in a Codex sandbox with network off, build and test against saved fixture files and ask before live runs.

Feed monitor (Python)

import hashlib, sqlite3, time
import feedparser

UA = "acme-feedwatch/1.0 (+mailto:[email protected])"
db = sqlite3.connect("feedwatch.db")
db.executescript("""
CREATE TABLE IF NOT EXISTS feeds(url TEXT PRIMARY KEY, etag TEXT, modified TEXT,
  last_ok REAL, failures INTEGER DEFAULT 0);
CREATE TABLE IF NOT EXISTS seen(feed TEXT, item_key TEXT, first_seen REAL,
  PRIMARY KEY(feed, item_key));
""")

def item_key(e) -> str:
    # guid/id first; fall back to link; last resort a hash of title+link
    raw = e.get("id") or e.get("link") or (e.get("title", "") + e.get("link", ""))
    return hashlib.sha256(raw.strip().encode()).hexdigest()

def check(url: str) -> list[dict]:
    row = db.execute("SELECT etag, modified, last_ok FROM feeds WHERE url=?", (url,)).fetchone()
    etag, modified, last_ok = row if row else (None, None, None)
    d = feedparser.parse(url, etag=etag, modified=modified, agent=UA)
    status = getattr(d, "status", None)
    if status == 304:
        return []
    if status is None or status >= 400 or (d.bozo and not d.entries):
        db.execute("INSERT INTO feeds(url, failures) VALUES(?,1) ON CONFLICT(url) "
                   "DO UPDATE SET failures = failures + 1", (url,))
        db.commit()
        raise RuntimeError(f"{url}: status={status} bozo={d.get('bozo_exception')}")
    first_run = last_ok is None                  # never succeeded before
    new = []
    for e in d.entries:
        key = item_key(e)
        cur = db.execute("INSERT OR IGNORE INTO seen VALUES(?,?,?)", (url, key, time.time()))
        if cur.rowcount and not first_run:      # first run seeds state silently
            new.append({"title": e.get("title"), "link": e.get("link"),
                        "published": e.get("published")})
    db.execute("INSERT INTO feeds VALUES(?,?,?,?,0) ON CONFLICT(url) DO UPDATE SET "
               "etag=excluded.etag, modified=excluded.modified, last_ok=excluded.last_ok, failures=0",
               (url, d.get("etag"), d.get("modified"), time.time()))
    db.commit()
    return new

Design points: the first run seeds state instead of alerting on fifty old posts; state is committed only after the item is recorded; failures are counted so you can alert on a broken feed, not just on new items.

Page change monitor

import difflib, hashlib, re
import httpx
from bs4 import BeautifulSoup

def snapshot(url: str, selector: str) -> str:
    r = httpx.get(url, headers={"User-Agent": UA}, timeout=30, follow_redirects=True)
    r.raise_for_status()
    node = BeautifulSoup(r.text, "lxml").select_one(selector)
    if node is None:
        raise RuntimeError(f"selector {selector!r} matched nothing on {url}; layout changed?")
    for junk in node.select("script, style, time, .ad, [data-timestamp]"):
        junk.decompose()
    text = node.get_text("\n", strip=True)
    text = re.sub(r"\b\d+ (minutes?|hours?|days?) ago\b", "", text)   # relative times
    return "\n".join(line for line in text.splitlines() if line.strip())

def changed(old: str, new: str) -> str | None:
    if hashlib.sha256(old.encode()).digest() == hashlib.sha256(new.encode()).digest():
        return None
    return "\n".join(difflib.unified_diff(old.splitlines(), new.splitlines(),
                                          "before", "after", lineterm="", n=1))

Store the last snapshot text per (url, selector) and send the diff, not just “page changed”. For lists, compare sets of item links: new_links = current - previous gives clean alerts even when the list reorders.

Noise filters that matter

Noise	Filter
Timestamps, “x minutes ago”, view counters	Strip by selector or regex before hashing
Rotating ads, recommendations	Narrow the selector to the content container
Feed items re-published with new updated dates	Key on id/link, not on date or content hash
Tracking parameters in links	Drop utm_*, fbclid, ref before comparing
Whitespace and markup changes	Compare extracted text, not HTML
A/B tests and geo variants	Pin Accept-Language; require the change to persist on two consecutive checks

Notifications

• Batch: one digest per run with all new items beats one message per item.
• Include title, link and the diff excerpt; cap message length.
• Sending to email, Slack or chat is outward-facing: confirm the destination with the user at setup and send a test message to them first. Webhook URLs and SMTP passwords come from environment variables, never from code.
• Alert on the monitor’s own health: a feed failing for more than a day, or a selector that stopped matching, is a notification too.

Off-the-shelf option

For many pages with a UI, self-host changedetection.io (Docker image ghcr.io/dgtlmoon/changedetection.io). It supports CSS/XPath filters, ignore-text rules, browser-based fetching for JS pages and many notification targets via Apprise. Recommend it when the user wants to manage watches by hand rather than in code. Running containers needs Docker and network access; ask first.

Scheduling

Run the checker from cron, a systemd timer, launchd or a scheduled GitHub Actions workflow ($github-actions-automation). Keep the SQLite file on persistent storage; in Actions, commit the state file or use a cache, otherwise every run is a first run.

Gotchas

• feedparser sets bozo = 1 on many slightly malformed feeds that still parse; treat it as fatal only when there are no entries.
• Some feeds change every guid on each build; detect it (all items new on every run) and fall back to the link as key.
• Servers that ignore conditional headers return 200 every time; the item-key table still prevents duplicates, which is why both layers exist.
• Feeds often contain only the last 10 to 20 items; polling too rarely misses items on busy feeds. Size the interval to the publish rate.
• Redirects to a new feed URL (301) should update the stored URL; feedparser exposes d.href and d.status.
• Time zones in published vary; use published_parsed (UTC struct_time) when sorting.

Verification

• Run twice against a saved fixture: first run seeds, second run reports zero new
• Add one item to the fixture: exactly one alert
• Live run logs a 304 on the second request for servers that support it
• Break the selector deliberately: the monitor reports a health error, not a change
• Test notification reached the confirmed destination

Handoff

**Watching:** 14 feeds, 3 pages (list in watches.yaml)
**Interval:** feeds 30 min, pages hourly, via systemd timer feedwatch.timer
**State:** /var/lib/feedwatch/feedwatch.db (seen items, ETags, failure counts)
**Alerts:** hourly digest to #product-intel (tested), health alert after 6 failed checks
**Noise rules:** timestamps stripped; price page compares .price-box only