Codex skills for LLM and AI apps
These skills cover the parts of an LLM app that tend to fail quietly: retrieval, prompts, evals, agents, streaming, caching, fine-tuning and serving. Codex measures before it tunes, using retrieval metrics, judges checked against human labels and eval gates in CI, and treats untrusted text and agent tools as security boundaries instead of problems for prompt wording.
Start with these 3
- LLM Evals Ship prompt changes on numbers, not vibes Codex builds eval suites from production samples, past bugs and adversarial inputs, grades with code wherever possible, and only trusts an LLM judge after measuring its agreement with human labels. A bundled standard-library gate compares runs case by case, reports per-tag shifts and fails CI when critical cases regress or the pass rate drops.
- LLM Observability Trace any answer back to prompt, context and cost Codex instruments LLM and RAG apps so each answer links to its prompt version, model, retrieved documents, tool calls, tokens, cost and latency. It uses Langfuse or OpenTelemetry GenAI conventions, masks sensitive fields before export, samples sensibly, flushes in short-lived jobs, and builds dashboards and alerts for cost per task, TTFT and thumbs-down rate.
- LLM Classification Labels you can trust, with confidence that means something Codex starts with the taxonomy, not the prompt: written definitions, precedence rules and an other label, checked against human agreement. Outputs are constrained to the label enum, confidence comes from logprobs or self-consistency and is calibrated on a validation split, and a bundled script reports per-class F1, the confusion matrix and ECE so prompt changes can be gated in CI.
30 more LLM and AI apps skills
-
LLM Streaming
Tokens that arrive live and stop when asked -
OpenAI Fine-Tuning
Fine-tune GPT-4.1 only when evals prove it pays -
LoRA Fine-Tuning
QLoRA a 7B model on one GPU and prove it helped -
Skill Evals
Prove a skill triggers when it should, and only then -
OpenAI Embeddings
One vector space, measured, never mixed -
Prompt Engineering
Versioned prompts, changed one eval at a time -
RAG Pipelines
Grounded answers with citations that resolve -
RAG Chunking
Chunks that follow structure and stand alone -
RAG Evaluation
Know whether retrieval or generation failed -
LlamaIndex RAG
Incremental ingestion and answers that show sources -
Ollama Local LLMs
Local models with the context you think you have -
Prompt Caching and Cost
Lower the LLM bill without lowering quality -
Semantic Caching
Cut LLM spend without serving wrong answers -
Prompt Injection Defense
Assume the text is hostile; make it not matter -
Hybrid Search and Reranking
BM25 plus vectors, fused by rank and proven on qrels -
Vector Databases
Pick the store from numbers, then plan the re-embed -
pgvector Search
HNSW that uses the index and returns a full page -
LangChain Apps
LangChain 1.x chains and agents that stay testable -
Multi-Agent Orchestration
More agents only when the eval says it pays -
Agent Memory
Remember what matters, forget on request -
Structured Extraction
JSON from messy text, with evidence and per-field scores -
vLLM Serving
Size KV cache from traffic, then benchmark it -
AI Chat Interface
Chat UIs that stream, stop and recover cleanly -
Synthetic Data Generation
Test and eval data with coverage you chose -
Model Quantization
Smaller, faster models with the quality loss measured -
AI Agent Evaluation
Judge agents by end state, reliability and cost -
Skill Authoring
From repeated prompt to a skill that triggers -
Skill Description Tuning
Make skills fire on cue, and only then -
LangGraph Agents
State graphs that pause, resume and survive crashes -
Agent Tool Design
Tools a model picks right and calls right first time
General skills with LLM and AI apps guidance
Not built around LLM and AI apps, but they include steps or tools for it. Not part of the install-all command.
-
libSQL and Turso
Local-speed reads with a remote SQLite primary -
OpenAI Batch API
Half-price bulk inference that rejoins cleanly -
Vercel AI SDK
Streaming chat and tools that match your ai version -
Document AI and OCR
Route each page, keep confidences, review the doubtful -
Hugging Face Transformers
Load, prompt and batch Hub models without surprises -
llama.cpp and GGUF
Pick the quant by perplexity, not folklore -
AI Content Moderation
Policy first, thresholds second, humans in the loop -
Form Data Extraction
Invoice fields as validated JSON, not guesses -
Project Management
Plans, RACI and status that match reality
Every LLM and AI apps skill in one command
Installs these 33 into ~/.agents/skills/. Add --project after -- to install into the current repository instead.
The installer is a 104-line POSIX shell script. It fetches the skills you name from codexskills.dev and unpacks each into its own folder in your skills directory. No sudo, no telemetry, and every skill page shows the full SKILL.md first. Read install.sh
curl -fsSL codexskills.dev/install.sh | sh -s -- llm-evals llm-observability llm-classification llm-streaming openai-fine-tuning lora-fine-tuning skill-evals openai-embeddings prompt-engineering rag-pipelines rag-chunking rag-evaluation llamaindex-rag ollama-local-llms prompt-caching-cost semantic-caching prompt-injection-defense hybrid-search-reranking vector-databases pgvector-search langchain-apps multi-agent-orchestration agent-memory structured-extraction vllm-serving ai-chat-interface synthetic-data-generation model-quantization ai-agent-evaluation skill-authoring skill-description-tuning langgraph-agents agent-tool-design LLM and AI apps FAQ
Is there a Codex skill for evaluating a RAG pipeline?
RAG Evaluation builds a frozen test set with graded relevance labels plus unanswerable and multi-hop questions, then scores retrieval and generation separately so each failure lands on a stage. Its bundled scripts/retrieval_metrics.py reports hit rate, recall, MRR and nDCG per k and per tag, with an optional recall gate. LLM judges are trusted only after they agree with human labels.
Do these skills support LangChain 1.x and LangGraph?
LangChain Apps targets LangChain 1.x on Python 3.10 or later, checks installed versions with pip show before trusting any snippet, uses create_agent for agent loops, and migrates legacy LLMChain and RetrievalQA code one chain at a time behind golden-output tests. LangGraph Agents builds state graphs with Postgres checkpointers keyed by a domain thread ID and interrupt-based approval in front of risky tools. Nodes are kept idempotent because they re-run on resume.
How does Codex defend an LLM app against prompt injection?
Prompt Injection Defense maps every LLM call for the combination of private data, untrusted content and outbound channels, then removes one leg or adds a deterministic gate rather than relying on prompt wording. It scopes tools per request, blocks auto-loading markdown images and links to unlisted domains, and ships an injection regression suite that asserts on effects. Agent Tool Design puts destructive tools behind human confirmation.
How do I install all 33 LLM and AI apps skills at once?
Pass every name to the installer in one command: curl -fsSL https://codexskills.dev/install.sh | sh -s -- llm-evals llm-observability llm-classification llm-streaming openai-fine-tuning lora-fine-tuning skill-evals openai-embeddings prompt-engineering rag-pipelines rag-chunking rag-evaluation llamaindex-rag ollama-local-llms prompt-caching-cost semantic-caching prompt-injection-defense hybrid-search-reranking vector-databases pgvector-search langchain-apps multi-agent-orchestration agent-memory structured-extraction vllm-serving ai-chat-interface synthetic-data-generation model-quantization ai-agent-evaluation skill-authoring skill-description-tuning langgraph-agents agent-tool-design. Add --project right after -- to put them in the current repository's .agents/skills/ so your team gets them too.
Which LLM and AI apps skill should I try first?
LLM Evals is the closest fit for most LLM and AI apps work. Mention it with $llm-evals in a prompt to load it on purpose; after that, Codex picks it up on its own whenever a task matches.