Speech to Text skill for Codex
Transcripts with speakers, timestamps and a WER. Codex picks the transcription engine from what you need: hosted accuracy, speaker labels, word timestamps, or audio that never leaves the machine. It normalizes audio with ffmpeg, chunks long recordings on silence, biases spelling of names with a short prompt, converts segments to readable captions with a bundled script, and reports WER on a hand-checked sample.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- speech-to-textInstalls to ~/.agents/skills/speech-to-text/ for every project on this machine.
curl -fsSL codexskills.dev/cli.mjs -o codex-skills.mjsnode codex-skills.mjs install speech-to-textThe CLI also searches, updates and removes skills. Run it with no arguments to pick from a list. Needs Node 20.12+.
curl -fsSL codexskills.dev/install.sh \ | sh -s -- --project speech-to-textInstalls to .agents/skills/speech-to-text/ in the current repository. Commit it and everyone who runs Codex there gets it.
curl -fsSLO codexskills.dev/downloads/speech-to-text.zipunzip -o speech-to-text.zip -d ~/.agents/skills/The archive holds the skill folder with its generated agents/openai.yaml. Unzip into .agents/skills/ to scope it to a repo.
& ([scriptblock]::Create((irm codexskills.dev/install.ps1))) speech-to-textPowerShell 5.1 or 7. Installs to %USERPROFILE%\.agents\skills\. Add -Project for the current repo.
104-line POSIX script. No sudo, no telemetry. Read install.sh
Use it in Codex
Start a prompt with $speech-to-text, or pick it from /skills.
You can also just describe the task: Codex reads the description below and loads the playbook when your request matches.
What Codex reads before loading it
Builds transcription pipelines with OpenAI speech-to-text (gpt-4o-transcribe, gpt-4o-mini-transcribe, gpt-4o-transcribe-diarize, whisper-1) and local Whisper via faster-whisper or whisper.cpp - ffmpeg audio prep, chunking long files under the 25 MB limit, word and segment timestamps, speaker diarization, vocabulary prompts, SRT/VTT subtitles, and WER measurement. Use when asked to transcribe audio or video, add subtitles or captions, label who said what, transcribe offline or on-device, or improve transcription accuracy on names and jargon.
Use it when
- Audio or video needs a transcript, captions or subtitles
- Speakers must be identified in meetings, interviews or calls
- Transcription must run offline or on-device for privacy
- Names and jargon keep coming out misspelled
Use something else when
- You are building a live voice agent that listens and replies: use openai-realtime-voice
- You need to cut, mix or convert audio files without transcribing: use audio-processing
How to check the result
WER on a hand-corrected sample of representative audio is reported per condition, and chunk boundaries show no dropped or duplicated words.
The mistake it prevents
Cutting long audio at fixed intervals instead of on silence, which drops or doubles words at every seam.
Done well means: audio is normalized before transcription, the model is chosen for the need (accuracy, timestamps, speakers, privacy, cost), long files are chunked on silence with no lost or duplicated words at boundaries, and accuracy is measured with WER on a sample against a human reference.
Choose the engine
| Need | Default |
|---|---|
| Best hosted accuracy, plain transcript | gpt-4o-transcribe (cheaper: gpt-4o-mini-transcribe) |
| Speaker labels from the API | gpt-4o-transcribe-diarize with response_format="diarized_json" |
| Word-level timestamps from the API | whisper-1 with verbose_json + timestamp_granularities |
| Audio must not leave the machine, or no network | faster-whisper (GPU/CPU) or whisper.cpp (CPU, Apple Silicon) |
| Word timestamps + diarization locally | WhisperX (alignment) + pyannote diarization |
| Live captions while someone speaks | Realtime transcription ($openai-realtime-voice) |
Audio of people is personal data. Confirm the user may process it, and with a third-party API, that participants consented and the data policy allows it. Hosted APIs need network; in a Codex sandbox ask before uploading anything.
Prepare audio
ffprobe -hide_banner input.mp4 # duration, codecs, channels
ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a pcm_s16le clean.wav # local models
ffmpeg -i input.mp4 -vn -ac 1 -ar 16000 -c:a libopus -b:a 32k clean.ogg # small upload- Mono 16 kHz is what Whisper-family models consume; higher rates only add size.
- Opus at 24-32 kbps keeps a 2-hour meeting well under 25 MB.
- Stereo call recordings with one speaker per channel: transcribe each channel separately
(
-af "pan=mono|c0=c0") for free, perfect diarization. - Loudness normalize very quiet recordings (
-af loudnorm); do not apply aggressive noise reduction, which often hurts accuracy.
OpenAI API
from openai import OpenAI
client = OpenAI()
with open("clean.ogg", "rb") as f:
tx = client.audio.transcriptions.create(
model="gpt-4o-transcribe",
file=f,
language="en", # skip auto-detect errors
prompt="Speakers discuss Kubernetes, Istio, and the Acme Payments API.",
)
print(tx.text)promptbiases spelling of names and jargon; keep it short and factual. Not supported by the diarize model.- File limit is 25 MB per request. Longer audio: split on silence (below) and pass the
previous chunk’s last sentence as
promptfor continuity. response_format:json/textfor gpt-4o models;srt,vtt,verbose_jsonforwhisper-1.
Word timestamps (whisper-1):
tx = client.audio.transcriptions.create(
model="whisper-1", file=open("clean.ogg", "rb"),
response_format="verbose_json", timestamp_granularities=["word", "segment"],
)
words = [(w.word, w.start, w.end) for w in tx.words]Diarization:
tx = client.audio.transcriptions.create(
model="gpt-4o-transcribe-diarize", file=open("meeting.ogg", "rb"),
response_format="diarized_json", chunking_strategy="auto",
)
for seg in tx.segments:
print(seg.speaker, seg.start, seg.end, seg.text)Set chunking_strategy for anything over about 30 seconds. To map speakers to names,
pass up to four 2-10 second reference clips as known_speaker_names and
known_speaker_references (via extra_body if the SDK version lacks the parameters).
Local Whisper
from faster_whisper import WhisperModel
model = WhisperModel("large-v3", device="cuda", compute_type="float16")
# CPU: device="cpu", compute_type="int8"; faster option: "large-v3-turbo" or "distil-large-v3"
segments, info = model.transcribe("clean.wav", language="en", beam_size=5,
vad_filter=True, word_timestamps=True,
initial_prompt="Kubernetes, Istio, Acme Payments")
for s in segments: # generator: iterate to actually run
print(f"[{s.start:.2f} -> {s.end:.2f}] {s.text}")whisper.cpp for CPU/Metal: build, fetch a ggml model, then
./build/bin/whisper-cli -m models/ggml-large-v3-turbo.bin -f clean.wav -l en -osrt -ovtt.
Model downloads need network once; then cache them (HF_HOME) for offline runs.
Chunking long audio
- Detect silences:
ffmpeg -i clean.wav -af silencedetect=noise=-35dB:d=0.6 -f null - 2>&1 | grep silence_end. - Cut at the silence nearest each target length (about 10 minutes, under 25 MB).
- Transcribe chunks (parallel is fine), then offset each chunk’s timestamps by its start time.
- Never cut mid-word on fixed boundaries; that drops or doubles words at every seam.
Subtitles
Turn segments or words into SRT/VTT with readable line lengths (bundled, stdlib only):
python3 scripts/segments_to_subtitles.py segments.json --format srt --max-chars 42 --max-duration 6 > out.srt
python3 scripts/segments_to_subtitles.py segments.json --format vtt > out.vttInput is a JSON list of {"start", "end", "text"} (or an object with segments or
words, as returned by the APIs above). Captions are split at 42 characters per line,
2 lines, and 6 seconds maximum; it exits non-zero on overlaps or invalid times.
Measuring accuracy
- Hand-correct 10-15 minutes of representative audio as the reference.
- WER = (substitutions + deletions + insertions) / reference words, after normalizing case,
punctuation and numbers (
jiwerwith a normalizing transform). - Report WER per condition (clean vs phone audio, accents) and the error on key terms (product names), which matters more than the overall rate.
Gotchas
- Whisper hallucinates text on silence or music (“Thanks for watching”). Use VAD
(
vad_filter=True) and drop segments with very low average log-probability. - Auto language detection on the first 30 seconds fails on code-switched or late-starting
audio; set
languagewhen known. - Speaker labels are arbitrary (
A,SPEAKER_00) and not stable across chunks unless known-speaker references are used; reconcile before showing names. - Timestamps drift after concatenation if chunk offsets use planned instead of actual cut times.
- Do not keep raw recordings longer than the user’s retention policy; delete only after the user confirms.
Report template
**Input:** board-meeting.mp4 (1h52m, stereo 48 kHz) -> mono 16 kHz opus, 11 chunks on silence
**Engine:** gpt-4o-transcribe-diarize (consent confirmed by user) | language en
**Output:** transcript.md (speaker-labelled), board.srt (1,804 cues)
**Accuracy:** WER 6.8% on a 12-minute hand-checked sample; product names 2 errors / 41
**Review needed:** 03:12-03:40 overlapping speech, speakers B/C uncertainWhat the zip contains
Codex reads bundled files only when a step points to them. Every one is shown here, so you can review it before installing.
agents/openai.yaml
Gives the Codex app a display name, a one-line description and a starter prompt for this skill.
interface:
display_name: "Speech to Text"
short_description: "Transcripts with speakers, timestamps and a WER"
default_prompt: "$speech-to-text Transcribe this two-hour meeting recording and label who is speaking"scripts/segments_to_subtitles.py
#!/usr/bin/env python3
"""Convert transcription segments or words (JSON) into SRT or WebVTT captions.
Usage:
python3 segments_to_subtitles.py segments.json [--format srt|vtt] [--max-chars 42]
[--max-lines 2] [--max-duration 6]
Input: a JSON list of {"start": float, "end": float, "text"|"word": str}, or an object
with a "segments" or "words" list (OpenAI verbose_json/diarized_json, faster-whisper dumps).
Words are regrouped into captions; long segments are split by character budget with times
interpolated by character position. Writes to stdout. Exits 1 on invalid or overlapping times.
"""
import argparse
import json
import sys
import textwrap
def load_items(path):
with open(path, encoding="utf-8") as fh:
data = json.load(fh)
if isinstance(data, dict):
data = data.get("words") or data.get("segments") or []
items = []
for i, d in enumerate(data):
text = (d.get("text") if d.get("text") is not None else d.get("word", "")).strip()
start, end = float(d["start"]), float(d["end"])
speaker = d.get("speaker")
items.append({"start": start, "end": end, "text": text, "speaker": speaker, "i": i})
return items
def split_long(item, budget, max_duration):
"""Split one segment into words with times interpolated by character position, then regroup."""
words = item["text"].split()
if not words:
return []
total = sum(len(w) + 1 for w in words)
dur = item["end"] - item["start"]
synthetic, pos = [], 0
for w in words:
start = item["start"] + dur * pos / total
pos += len(w) + 1
synthetic.append({"start": start, "end": item["start"] + dur * pos / total,
"text": w, "speaker": item["speaker"]})
return group_words(synthetic, budget, max_duration)
def group_words(items, budget, max_duration):
cues, cur = [], []
for w in items:
text = " ".join(x["text"] for x in cur + [w])
too_long = len(text) > budget
too_slow = cur and w["end"] - cur[0]["start"] > max_duration
speaker_change = cur and w["speaker"] != cur[-1]["speaker"]
sentence_end = cur and cur[-1]["text"].endswith((".", "?", "!")) and len(text) > budget // 2
if cur and (too_long or too_slow or speaker_change or sentence_end):
cues.append(cur)
cur = []
cur.append(w)
if cur:
cues.append(cur)
return [{"start": c[0]["start"], "end": c[-1]["end"], "text": " ".join(x["text"] for x in c),
"speaker": c[0]["speaker"]} for c in cues]
def ts(seconds, sep):
ms = int(round(seconds * 1000))
h, ms = divmod(ms, 3_600_000)
m, ms = divmod(ms, 60_000)
s, ms = divmod(ms, 1000)
return f"{h:02d}:{m:02d}:{s:02d}{sep}{ms:03d}"
def main():
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
ap.add_argument("path")
ap.add_argument("--format", choices=["srt", "vtt"], default="srt")
ap.add_argument("--max-chars", type=int, default=42, help="characters per line")
ap.add_argument("--max-lines", type=int, default=2)
ap.add_argument("--max-duration", type=float, default=6.0, help="seconds per cue")
args = ap.parse_args()
try:
items = load_items(args.path)
except (OSError, ValueError, KeyError, TypeError) as exc:
print(f"error: cannot read input: {exc}", file=sys.stderr)
return 1
errors = [f"item {x['i']}: end before start" for x in items if x["end"] < x["start"]]
ordered = sorted(items, key=lambda x: x["start"])
for a, b in zip(ordered, ordered[1:]):
if b["start"] < a["end"] - 0.05 and a["speaker"] == b["speaker"]:
errors.append(f"items {a['i']} and {b['i']} overlap ({a['end']:.2f} > {b['start']:.2f})")
if errors:
for e in errors[:20]:
print(f"error: {e}", file=sys.stderr)
return 1
budget = args.max_chars * args.max_lines
is_words = all(len(x["text"].split()) <= 1 for x in ordered)
if is_words:
cues = group_words([x for x in ordered if x["text"]], budget, args.max_duration)
else:
cues = [c for x in ordered for c in split_long(x, budget, args.max_duration)]
out = ["WEBVTT", ""] if args.format == "vtt" else []
sep = "." if args.format == "vtt" else ","
for n, c in enumerate(cues, start=1):
text = c["text"]
if c["speaker"] and args.format == "vtt":
text = f"<v {c['speaker']}>{text}"
elif c["speaker"]:
text = f"[{c['speaker']}] {text}"
lines = textwrap.wrap(text, width=args.max_chars) or [""]
if args.format == "srt":
out.append(str(n))
out.append(f"{ts(c['start'], sep)} --> {ts(c['end'], sep)}")
out.extend(lines)
out.append("")
sys.stdout.write("\n".join(out))
print(f"{len(cues)} cues written", file=sys.stderr)
return 0
if __name__ == "__main__":
sys.exit(main())
About this skill
Written and maintained by the Codex Skills editors to the same editorial standard as every skill here: a selective description, a real workflow, known pitfalls and a way to verify the work. First published , last updated . MIT licensed. Something wrong or out of date? Tell us.