Document AI and OCR skill for Codex

Done well means: each page is routed by what it is (born-digital, scanned, photo, form), text comes out with reading order, tables and coordinates preserved, low-confidence regions are flagged instead of silently trusted, the LLM step works on clean structured text (or the page image when layout matters), and accuracy is measured on labelled pages.

Classify pages first

pdffonts in.pdf | head                    # no fonts listed => likely image-only
pdftotext -f 1 -l 3 -layout in.pdf - | head -50
pdfimages -list in.pdf | head             # page-sized images => scans
pdftoppm -r 150 -f 1 -l 1 -png in.pdf page   # look at it

Page type	Route
Born-digital PDF with good text layer	Skip OCR; extract text/tables ($pdf-processing) or layout-aware parse (Docling)
Scanned PDF, clean print	OCRmyPDF (Tesseract) or Docling with OCR
Phone photos, skewed, shadows	Preprocess (deskew, dewarp, crop) then OCR; or a vision LLM
Tables-heavy (statements, lab reports)	Layout model with table structure (Docling, Textract TABLES, Azure prebuilt-layout)
Fixed forms (tax forms, applications)	Key-value extraction (Textract FORMS, Azure prebuilt or custom model) or template coordinates
Receipts, invoices, IDs	Prebuilt specialised models (Textract AnalyzeExpense/AnalyzeID, Azure prebuilt-invoice/receipt, Google processors)
Handwriting	Cloud APIs or vision LLMs; Tesseract is weak here
Mixed PDFs (some pages scanned)	Per-page routing; never OCR pages that already have good text

Engine choice

Need	Default	Notes
Local, offline, printed text	OCRmyPDF + Tesseract	Adds a searchable text layer; mature CLI
Local, layout + tables to Markdown/JSON	Docling	Reading order, tables, figures; uses OCR when needed
Local, multilingual/CJK, better on photos	PaddleOCR	Heavier install, GPU helps
Managed, forms/tables/receipts at scale	Textract, Azure Document Intelligence, Google Document AI	Per-page pricing; data leaves your network
Messy layouts, semantic questions about the page	Vision-capable LLM on page images	Costly per page; can hallucinate unreadable text

Data residency and confidentiality decide between local and cloud before accuracy does; ask the user if documents may be sent to a cloud API. Installing engines, downloading models and calling cloud APIs need network; in a Codex sandbox with network off, request approval or use what is already installed (tesseract --version, ocrmypdf --version).

Workflow

- [ ] 1. Sample 30-50 pages across sources; label ground truth for the fields that matter
- [ ] 2. Classify pages (table above) and route per page
- [ ] 3. Preprocess images where needed (300 DPI, deskew, contrast)
- [ ] 4. OCR/layout -> structured intermediate (Markdown or JSON with page, bbox, confidence)
- [ ] 5. LLM extraction over the intermediate (or page image) with page references
- [ ] 6. Validate; route low-confidence and failed checks to human review
- [ ] 7. Measure field accuracy and character error rate on the labelled sample; iterate

Local OCR

# Searchable PDF; keeps existing text pages, OCRs image pages
ocrmypdf --skip-text --rotate-pages --deskew -l eng+deu --output-type pdf in.pdf out.pdf
pdftotext -layout out.pdf out.txt

# Single image, with per-word confidences as TSV
tesseract receipt.png receipt -l eng --psm 6 tsv

• --skip-text leaves pages with text untouched; --redo-ocr replaces an existing bad OCR layer; --force-ocr rasterises everything (loses vector text quality, use sparingly).
• --psm 6 (uniform block) suits receipts; --psm 4 single column of varying sizes; default 3 for full pages. Wrong page segmentation mode is a common cause of garbage.
• Install language packs for every language present (tesseract --list-langs).

from docling.document_converter import DocumentConverter

result = DocumentConverter().convert("statement.pdf")
markdown = result.document.export_to_markdown()     # headings, reading order, tables as Markdown
doc_json = result.document.export_to_dict()         # elements with page numbers and bounding boxes

Feed Markdown tables to the LLM rather than flattened text; table structure survives.

Image preprocessing (photos and poor scans)

• Render PDFs at 300 DPI for OCR (pdftoppm -r 300); 150 DPI loses small print.
• Deskew and rotate (OCRmyPDF does this; OpenCV minAreaRect on text pixels for custom code).
• Convert to grayscale; apply adaptive thresholding only if it measurably helps; it can erase faint text.
• Crop to the document (contour detection) for phone photos; perspective-correct before OCR.
• Keep the original image; store preprocessing parameters with the result for debugging.

Cloud document APIs

• Textract: AnalyzeDocument with FeatureTypes=["TABLES","FORMS"] (add QUERIES for targeted questions), AnalyzeExpense for receipts/invoices; multi-page PDFs use the async Start.../Get... APIs via S3.
• Azure Document Intelligence: prebuilt-layout for text/tables/structure, prebuilt-invoice, prebuilt-receipt, prebuilt-idDocument, or a custom model trained on your forms.
• Google Document AI: create a processor (OCR, Form Parser, Invoice, custom extractor) in the project, then call it by processor ID.
• All return confidences per field or word: keep them in your intermediate format; they drive the review queue.
• Respect quotas with backoff; record page counts for cost tracking.

LLM step

Two ways to combine OCR with an LLM:

Approach	Use when
OCR/layout text (Markdown) -> text LLM with a schema	Clean print, tables, many pages; cheaper and auditable
Page image (+ optional OCR text) -> vision LLM	Complex layouts, checkboxes, stamps, handwriting, charts

Rules:

• Include page numbers in the input (<page n="3">...</page>) and ask for page per value.
• Ask for verbatim evidence for key fields and check it against the OCR text in code.
• For vision LLMs, tell the model to return null for unreadable regions; verify numbers that matter against OCR output or a second pass, because vision models can produce plausible-looking digits.
• Schema design, validation and accuracy scoring: $structured-extraction.

Human review

Route to review when any holds: OCR confidence on a critical field below threshold (start at 0.85 and tune), validation failure (totals, checksums, dates), evidence quote not found, page classified as handwriting or low quality. Show the reviewer the cropped region (bbox), the OCR text and the proposed value side by side, and record corrections as new gold labels.

Gotchas

• OCRing born-digital pages lowers quality; always check for a text layer first.
• Multi-column pages: plain OCR text interleaves columns; use layout-aware output (Docling, prebuilt-layout, Tesseract --psm tuned) and verify reading order on samples.
• Tables spanning pages lose headers on continuation pages; carry the header forward in code.
• Numbers: OCR confuses 0/O, 1/l/I, 5/S, ,/.; validate numeric fields with regex and checksums (IBAN, VAT IDs, totals).
• IDs and medical or financial documents are sensitive: keep processing in the approved region, avoid logging page text, and delete intermediates per the retention rule agreed with the user.
• Very large batches: process page by page, persist per-page results, make reruns resumable.

Verification

• Character error rate (CER) on 20+ labelled pages per source type, compared across engines before choosing one.
• Field accuracy on the labelled sample meets the threshold ($structured-extraction script).
• Review-queue rate and reviewer correction rate tracked weekly; corrections become gold data.
• Spot-check 5 pages per new source visually: overlay bounding boxes on the image.

Report template

**Corpus:** 1,840 PDFs (62% born-digital, 31% scanned, 7% phone photos)
**Routing:** text layer -> Docling; scans -> OCRmyPDF --skip-text --deskew; photos -> crop + Textract AnalyzeExpense
**Accuracy (60 labelled pages):** CER 1.8% scans, 4.9% photos; total field acc 0.98, date 0.95
**Review:** 9% of documents (low confidence 6%, sum mismatch 3%)
**Data handling:** processed in eu-west-1, no page text in logs, intermediates deleted after 30 days