Ollama Local LLMs skill for Codex

Done well means: the model tag and quantization fit the machine with the context the task needs, the context length is set explicitly (never assumed), calls are reproducible, and the server is not exposed beyond localhost without protection.

First inspect

ollama --version
ollama list                    # installed models and sizes
ollama ps                      # loaded models, size, CPU/GPU split, context, expiry
ollama show qwen2.5:7b         # architecture, context length, quantization, parameters, template
nvidia-smi || system_profiler SPDisplaysDataType | head -20

Read AGENTS.md for which model and host the project expects. ollama pull needs network; in a Codex sandbox with network off, ask for approval or use models already listed. The API listens on http://127.0.0.1:11434; a sandbox may also block localhost sockets, so ask before assuming the server is unreachable because it is down.

Pick a model and quantization

Machine	Fits comfortably
8 GB RAM/VRAM	3-4B models at Q4_K_M
16 GB	7-8B at Q4_K_M or Q8_0 with moderate context
24 GB VRAM / 32 GB unified memory	12-14B at Q4_K_M, or 7-8B at Q8_0 with long context
48 GB+	30-32B at Q4_K_M; 70B at Q4 needs about 40 GB plus KV cache

• Default tags are usually Q4_K_M. Pick explicit tags such as qwen2.5:7b-instruct-q8_0 when quality matters more than speed.
• KV cache grows with context: doubling num_ctx can push layers to CPU. Check the PROCESSOR column of ollama ps; “100% GPU” is the goal.
• Embeddings need an embedding model (nomic-embed-text, mxbai-embed-large, qwen3-embedding and similar), not a chat model.

Context length: always set it

Ollama’s default context window depends on version and available memory and is often far below the model’s maximum. Long prompts are truncated from the start without an error.

OLLAMA_CONTEXT_LENGTH=16384 ollama serve     # server-wide default

Per request, set "options": {"num_ctx": 16384}, or bake PARAMETER num_ctx into a Modelfile. Confirm with ollama ps (context column in recent versions) or by checking prompt_eval_count in the response against the expected token count.

Native API

curl http://127.0.0.1:11434/api/chat -d '{
  "model": "qwen2.5:7b",
  "messages": [{"role": "user", "content": "Name three HTTP idempotent methods."}],
  "stream": false,
  "options": {"temperature": 0, "num_ctx": 8192, "seed": 42},
  "keep_alive": "30m"
}'

from ollama import chat, embed  # pip install ollama

resp = chat(
    model="qwen2.5:7b",
    messages=[{"role": "user", "content": "Extract the city: 'Ship to 12 Rue Cler, Paris'"}],
    format={"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]},
    options={"temperature": 0},
)
print(resp.message.content)            # JSON string matching the schema

vecs = embed(model="nomic-embed-text", input=["first doc", "second doc"]).embeddings

• format accepts "json" or a JSON Schema; still validate the result (Pydantic model_validate_json) because small models can produce valid JSON with wrong values.
• Tool calling: pass tools=[...] with JSON-schema functions; read resp.message.tool_calls, run them, append {"role": "tool", ...} and call again. Only models whose template supports tools will call them (ollama show lists capabilities).
• Reasoning models accept think=True and return thinking separately from content.
• Streaming returns newline-delimited JSON objects; the last has "done": true plus timing stats.

OpenAI-compatible endpoint

from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:11434/v1", api_key="ollama")  # key ignored
r = client.chat.completions.create(model="qwen2.5:7b", messages=[{"role": "user", "content": "hi"}])

Use it to swap a hosted model for a local one in existing code. Options without an OpenAI equivalent (num_ctx) cannot be passed here; set them in a Modelfile or server env instead.

Modelfiles

FROM qwen2.5:7b
PARAMETER num_ctx 16384
PARAMETER temperature 0.2
PARAMETER stop "<|im_end|>"
SYSTEM """You are the support assistant for Acme. Answer from the provided context only."""

ollama create acme-support -f Modelfile
ollama run acme-support "How do I reset my password?"
ollama show acme-support --modelfile          # inspect what was built

Import a GGUF (for example a merged fine-tune from $llama-cpp-gguf): FROM ./model-Q4_K_M.gguf. Copy the TEMPLATE block from ollama show <similar-model> --modelfile if the GGUF lacks an embedded chat template; a wrong template causes rambling and ignored system prompts. ADAPTER ./lora-adapter applies a LoRA to a matching base.

Server configuration

Env var	Purpose
OLLAMA_HOST	Bind address, default 127.0.0.1:11434
OLLAMA_KEEP_ALIVE	How long models stay loaded (5m default; -1 forever)
OLLAMA_NUM_PARALLEL	Concurrent requests per model (each multiplies KV memory)
OLLAMA_MAX_LOADED_MODELS	Models resident at once
OLLAMA_FLASH_ATTENTION=1	Lower memory for long context on supported GPUs
OLLAMA_KV_CACHE_TYPE=q8_0	Quantized KV cache (needs flash attention) to fit more context
OLLAMA_MODELS	Where model blobs live

On Linux with systemd, set these with systemctl edit ollama under [Service] as Environment=..., then restart; editing shell profiles does not affect the service. On macOS app installs, use launchctl setenv and restart the app.

Exposing Ollama: the API has no authentication. Before setting OLLAMA_HOST=0.0.0.0, confirm with the user and put it behind a reverse proxy with auth and TLS, or a VPN/tailnet. Never open port 11434 to the internet.

Troubleshooting

Symptom	Cause / fix
Answers ignore early parts of a long prompt	Context truncated: raise num_ctx
First request slow, later fast	Model load; raise keep_alive or warm it with an empty request
ollama ps shows CPU/GPU split	Model + KV cache exceed VRAM: smaller quant, smaller context, or fewer parallel slots
Gibberish or endless output from imported GGUF	Wrong or missing TEMPLATE / stop tokens
Different answers on identical requests	Set temperature: 0 and seed; parallel batching can still vary slightly
Connection refused from a container	Use host.docker.internal:11434 or bind to the bridge IP

Verification

• ollama ps shows the model on GPU with the expected context.
• Response prompt_eval_count matches the prompt size (no truncation) and eval_count / eval_duration gives tokens per second for a fixed prompt.
• Structured calls validate against the schema on 20 sample inputs.

Report template

**Model:** qwen2.5:7b-instruct-q8_0 (8.1 GB) via Modelfile `acme-support`
**Context:** num_ctx 16384, confirmed in ollama ps; 100% GPU on RTX 4090
**Speed:** 92 tok/s generation, 1.4 s time to first token (warm)
**API:** /api/chat with JSON schema; 20/20 outputs validated
**Exposure:** bound to 127.0.0.1; remote access via tailnet only