Hugging Face Transformers skill for Codex

Done well means: the model loads in the intended dtype on the intended device, prompts go through the model’s chat template, generation settings are explicit and reproducible, batches are padded on the correct side, and the code runs offline from a pinned revision.

Before writing code

1. Read AGENTS.md and the lockfile; note the installed transformers and torch versions (python -c "import transformers, torch; print(transformers.__version__, torch.__version__)"). Transformers v5 removed TensorFlow/Flax support and uses dtype= (older code says torch_dtype=); match the installed version.
2. Read the model card: license, gated access, expected prompt format, context length, and whether it needs trust_remote_code=True (review that code before enabling it).
3. Check hardware: CUDA, Apple mps, or CPU only. That decides dtype and model size.
4. Downloads need network. In a Codex sandbox with network off, use a populated cache: HF_HOME=/path/to/cache HF_HUB_OFFLINE=1.

Choose the entry point

Need	Use
Quick, standard task (classification, NER, summarization, ASR)	pipeline(task, model=...)
Chat or instruction model, custom decoding	AutoTokenizer + AutoModelForCausalLM + generate()
Sentence embeddings for search	sentence-transformers ($openai-embeddings for hosted)
High-throughput serving, many concurrent users	Not Transformers: $vllm-serving
CPU or laptop with quantized weights	$llama-cpp-gguf or $ollama-local-llms

Device and dtype

Hardware	dtype	Load call
NVIDIA Ampere or newer (A100, L4, RTX 30/40/50)	torch.bfloat16	device_map="auto"
Older NVIDIA (T4, V100)	torch.float16	device_map="auto"
Apple Silicon	torch.float16	.to("mps")
CPU	torch.float32 (or bfloat16 on recent CPUs)	default
Model larger than VRAM	4-bit via BitsAndBytesConfig (CUDA)	device_map="auto"

Memory rule of thumb for weights alone: parameters x bytes (bf16 = 2, int8 = 1, 4-bit about 0.55). An 8B model is about 16 GB in bf16, plus KV cache that grows with batch x length.

Chat generation

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

model_id = "Qwen/Qwen2.5-7B-Instruct"
revision = "main"  # pin a commit hash in production

tok = AutoTokenizer.from_pretrained(model_id, revision=revision)
model = AutoModelForCausalLM.from_pretrained(
    model_id, revision=revision, dtype=torch.bfloat16, device_map="auto"
)
model.eval()

messages = [
    {"role": "system", "content": "Answer in one sentence."},
    {"role": "user", "content": "What does HTTP 409 mean?"},
]
inputs = tok.apply_chat_template(
    messages, add_generation_prompt=True, return_tensors="pt", return_dict=True
).to(model.device)

with torch.inference_mode():
    out = model.generate(
        **inputs, max_new_tokens=200, do_sample=False,
        pad_token_id=tok.pad_token_id or tok.eos_token_id,
    )
answer = tok.decode(out[0, inputs["input_ids"].shape[1]:], skip_special_tokens=True)

• Slice off the prompt tokens before decoding, or the answer includes the prompt.
• add_generation_prompt=True appends the assistant header; without it, many models continue the user turn.
• Greedy (do_sample=False) for extraction and tests; sampling (temperature=0.7, top_p=0.9) for creative text. Set torch.manual_seed when sampling must be reproducible.

Batching

tok.padding_side = "left"                     # decoder-only generation needs left padding
if tok.pad_token is None:
    tok.pad_token = tok.eos_token

prompts = [tok.apply_chat_template(m, add_generation_prompt=True, tokenize=False) for m in batch]
enc = tok(prompts, return_tensors="pt", padding=True, add_special_tokens=False).to(model.device)
out = model.generate(**enc, max_new_tokens=128, do_sample=False)
texts = tok.batch_decode(out[:, enc["input_ids"].shape[1]:], skip_special_tokens=True)

add_special_tokens=False because the chat template already inserted BOS; a doubled BOS quietly hurts quality. Sort prompts by length to cut padding waste.

Pipelines for standard tasks

from transformers import pipeline

clf = pipeline("text-classification", model="distilbert/distilbert-base-uncased-finetuned-sst-2-english",
               device_map="auto", truncation=True, max_length=512)
clf(["Great product", "Broke after a day"], batch_size=32)

ner = pipeline("token-classification", model="dslim/bert-base-NER", aggregation_strategy="simple")

Pass truncation=True for encoder models; inputs past max_position_embeddings otherwise error.

Speed and memory

Problem	Fix
Slow generation, GPU underused	Batch requests; use attn_implementation="sdpa" (default) or "flash_attention_2" if installed
OOM on load	4-bit/8-bit loading, smaller model, or device_map="auto" to offload (slow)
OOM during long generation	Lower max_new_tokens or batch size; KV cache dominates
Repeated warm-up cost in a script	Load once at module level; reuse the model object
Many users at once	Move to vLLM or TGI-style serving instead of tuning generate()

from transformers import BitsAndBytesConfig
q = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)
model = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=q, device_map="auto")

Offline and reproducible

hf download Qwen/Qwen2.5-7B-Instruct --revision <commit-sha>   # needs network; older hubs: huggingface-cli download
export HF_HOME=/models/hf HF_HUB_OFFLINE=1

• Pin revision= to a commit SHA; main can change under you.
• local_files_only=True in from_pretrained fails fast instead of trying the network.
• Store models outside the repo; never commit weights or HF_TOKEN.

Gotchas

• Warning “A decoder-only architecture is being used, but right-padding was detected”: set padding_side="left" before tokenizing.
• max_length counts prompt plus output; use max_new_tokens for generation length.
• generation_config.json on the Hub sets defaults (temperature, top_p); passing do_sample=False overrides sampling but warns about unused flags; that warning is harmless.
• trust_remote_code=True executes Python from the repo. Pin the revision and read the code.
• Tokenizer and model must come from the same repo and revision; mixing them misaligns vocab.
• Logits from different dtypes differ slightly; do not compare float outputs with == in tests.

Verification

• Print model.dtype, model.device (or model.hf_device_map) and peak memory (torch.cuda.max_memory_allocated() / 1e9).
• Run 3-5 fixed prompts greedily and store expected outputs as a smoke test.
• Time tokens/sec for a fixed batch before and after any speed change.

Output contract

**Model:** Qwen/Qwen2.5-7B-Instruct @ <sha> (Apache-2.0) | **dtype/device:** bf16, cuda:0
**Memory:** peak 17.2 GB | **Throughput:** 38 tok/s single, 410 tok/s at batch 16
**Decoding:** greedy, max_new_tokens=200 | **Offline:** HF_HUB_OFFLINE=1 verified