llama.cpp and GGUF skill for Codex

Done well means: the build uses the right GPU backend, the quant is chosen by measured quality loss and memory fit rather than habit, the chat template is correct, and speed is reported from llama-bench on the target machine.

Inspect first

git -C llama.cpp log -1 --oneline        # llama.cpp moves fast: record the commit
nvidia-smi || sysctl -n machdep.cpu.brand_string
free -g 2>/dev/null || sysctl hw.memsize
ls -la models/

Read AGENTS.md for where models live. Cloning, building dependencies and downloading weights need network; in a Codex sandbox ask for approval or reuse an existing build.

Build

git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
cmake -B build                           # CPU; Metal is on by default on Apple Silicon
cmake -B build -DGGML_CUDA=ON            # NVIDIA
cmake -B build -DGGML_VULKAN=ON          # AMD/Intel via Vulkan (or -DGGML_HIP=ON for ROCm)
cmake --build build --config Release -j
ls build/bin/                            # llama-cli, llama-server, llama-quantize, llama-bench, ...

Prebuilt release binaries and package managers (brew install llama.cpp) are fine for running; build from source when you need conversion scripts at the matching commit.

Convert a Hugging Face model

python -m pip install -r requirements.txt          # converter deps, needs network
python convert_hf_to_gguf.py /models/my-merged-model \
  --outfile /models/my-model-f16.gguf --outtype f16   # or bf16 to keep range exactly

• Convert a merged model (LoRA merged into the bf16 base, see $lora-fine-tuning); standalone adapters convert with convert_lora_to_gguf.py and load with --lora.
• The converter embeds the tokenizer and chat template from tokenizer_config.json. If the template is missing, generation rambles; fix the source config and reconvert.
• New architectures need a recent llama.cpp; “model architecture not supported” means update first.

Choose a quant

Quant	Bits/weight (approx)	Use when
Q8_0	8.5	Near-lossless reference; memory is not tight
Q6_K	6.6	High quality, modest saving
Q5_K_M	5.7	Good default when it fits
Q4_K_M	4.8	Common default for 7-70B on consumer hardware
IQ4_XS	4.3	Slightly smaller than Q4_K_M, needs imatrix for best results
Q3_K_M, IQ3_*, IQ2_*	3.9 and below	Only for very large models that do not fit otherwise; measure loss

File size is roughly parameters x bits / 8. Add KV cache (grows with context and parallel slots) and about 0.5-1 GB overhead to get the memory need.

# importance matrix from representative text (helps IQ and low-bit K quants)
./build/bin/llama-imatrix -m /models/my-model-f16.gguf -f calibration.txt -o /models/imatrix.gguf -ngl 99
./build/bin/llama-quantize --imatrix /models/imatrix.gguf \
  /models/my-model-f16.gguf /models/my-model-Q4_K_M.gguf Q4_K_M

Calibration text should resemble the real workload (code for a code model, the target languages for multilingual use). Older builds write imatrix.dat; both formats load.

Measure quality loss

./build/bin/llama-perplexity -m /models/my-model-f16.gguf -f eval.txt -ngl 99 -c 2048
./build/bin/llama-perplexity -m /models/my-model-Q4_K_M.gguf -f eval.txt -ngl 99 -c 2048

Use the same text and context for both. Perplexity increase under about 1-2% is usually invisible; above about 5%, try a larger quant or imatrix. Then run the task eval ($llm-evals) on the quant you will ship; perplexity does not catch every regression.

Run and serve

./build/bin/llama-cli -m /models/my-model-Q4_K_M.gguf -ngl 99 -c 8192 -cnv   # interactive chat

./build/bin/llama-server -m /models/my-model-Q4_K_M.gguf \
  --host 127.0.0.1 --port 8080 \
  -ngl 99 -c 32768 -np 4 \
  --jinja -fa on \
  -ctk q8_0 -ctv q8_0 \
  --api-key "$LLAMA_API_KEY"

Flag	Meaning and rule
-ngl N	Layers offloaded to GPU; 99 = all. Lower it until the model fits VRAM
-c N	Total context; with -np 4 each slot gets c / 4
-np N	Parallel slots (continuous batching)
--jinja	Use the GGUF’s chat template; needed for tool calling
-fa on	Flash attention (older builds: bare -fa); required for quantized V cache
-ctk/-ctv q8_0	Quantized KV cache, about half the memory of f16
-hf user/repo:Q4_K_M	Download a GGUF from the Hub directly (needs network)
--mlock / --no-mmap	Pin weights in RAM / load fully instead of memory-mapping

The server exposes /v1/chat/completions, /v1/completions, /v1/embeddings (with --embeddings), /health and /metrics (with --metrics). It also supports response_format JSON schemas and GBNF grammars for constrained output.

Binding to 0.0.0.0 exposes the model to the network: ask first, and keep an API key plus a proxy or VPN in front.

Benchmark

./build/bin/llama-bench -m /models/my-model-Q4_K_M.gguf -ngl 99 -p 512 -n 128 -fa 1

Report prompt processing (pp512) and generation (tg128) tokens/s. CPU generation speed is bound by memory bandwidth; set threads (-t) to physical cores and compare 2-3 values.

Gotchas

• GGUFs made with an old converter can lack newer metadata (rope scaling, templates); reconvert from the source model rather than patching.
• Quantizing an already-quantized GGUF compounds error. Always start from f16/bf16 (--allow-requantize exists, but avoid it).
• Context beyond the model’s trained length needs rope scaling settings and degrades quality; do not raise -c past the trained context casually.
• Partial offload (-ngl less than all layers) is much slower than full offload; a smaller quant fully on GPU often beats a larger one split.
• Flag spellings change between builds; check --help for the commit you built.

Report template

**Source:** org/model @ <sha> -> f16 GGUF (llama.cpp <commit>)
**Quant:** Q4_K_M with imatrix (calibration: 2 MB of support tickets), 4.6 GB
**Quality:** PPL 6.41 vs 6.29 f16 (+1.9%); task eval 88% vs 89% f16
**Speed (M3 Pro, Metal):** pp512 410 tok/s, tg128 34 tok/s
**Serving:** llama-server -c 32768 -np 4 --jinja -fa on, q8_0 KV, bound to 127.0.0.1