JVM Tuning skill for Codex

Done well means: the goal is explicit (pause target, throughput, memory footprint), the current behaviour is measured with GC logs and JFR under representative load, changes are few and justified, and the result is confirmed with the same load. Most good JVM configs are short: a collector choice, a heap size, logging and diagnostics.

Gather facts first

java -version                                    # JDK version and vendor
jcmd <pid> VM.flags                              # effective flags (incl. ergonomics)
jcmd <pid> VM.command_line
jcmd <pid> GC.heap_info
java -XX:+PrintFlagsFinal -version | grep -E 'UseG1GC|UseZGC|MaxHeapSize|MaxRAMPercentage|ActiveProcessorCount'

Also: container memory and CPU limits, pod restart reasons (OOMKilled = whole process exceeded the container limit, not necessarily the heap), current p50/p99 latency, and GC CPU share. Read AGENTS.md and the Dockerfile/Helm values for where JVM flags live (JAVA_TOOL_OPTIONS, JDK_JAVA_OPTIONS, start scripts).

Always-on diagnostics (cheap, add first)

-Xlog:gc*,safepoint:file=/var/log/app/gc.log:time,uptime,level,tags:filecount=5,filesize=20m
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/dumps
-XX:StartFlightRecording=disk=true,maxage=6h,settings=default,dumponexit=true,filename=/dumps/app.jfr
-XX:NativeMemoryTracking=summary        # small overhead; enable while investigating RSS

Heap dumps and JFR files contain application data; store them on a protected volume and treat them as sensitive.

Choose the collector

Goal / workload	Collector	Flag
General services, heaps 1–32 GB, pauses ~50–200 ms acceptable	G1 (default)	none, or -XX:+UseG1GC
p99 latency matters, pauses must be ~1 ms, larger heaps	Generational ZGC	-XX:+UseZGC (generational is the only mode from JDK 24; JDK 21–22 need -XX:+ZGenerational)
Low pause on OpenJDK builds that ship it	Shenandoah (generational mode available in recent JDKs)	-XX:+UseShenandoahGC
Batch jobs, maximum throughput, pauses irrelevant	Parallel	-XX:+UseParallelGC
Tiny containers (< ~1.7 GB or 1 CPU)	Serial is chosen by ergonomics	Give the container 2 CPUs, or set the collector explicitly

Low-pause collectors trade some throughput and memory headroom for latency; ZGC wants extra free heap to stay concurrent. Do not switch collectors to fix a leak or allocation storm; fix the code first ($memory-leak-hunting, $flamegraphs with async-profiler alloc mode).

Size memory in containers

• Prefer -XX:MaxRAMPercentage=70 (60–75) over fixed -Xmx, so the heap follows the container limit. Leave room for Metaspace, thread stacks (~1 MB each by default), code cache, direct buffers, GC structures and the libc allocator.
• For latency-sensitive services, set initial = max (-XX:InitialRAMPercentage equal to max, or -Xms = -Xmx) to avoid resize pauses, and consider -XX:+AlwaysPreTouch (slower startup, no page-fault stalls later).
• RSS above the container limit with a healthy heap is native memory:

jcmd <pid> VM.native_memory baseline
# ... wait under load ...
jcmd <pid> VM.native_memory summary.diff

Growth in Thread (too many threads), Internal/Other (direct buffers, Netty), Class/Metaspace (classloader leaks, dynamic proxies) points at the culprit. Cap -XX:MaxDirectMemorySize and -XX:MaxMetaspaceSize only after finding the cause.

• ActiveProcessorCount follows container CPU limits; with fractional or very low limits the JVM may pick a single-threaded collector and few GC threads.

Reading GC logs and JFR

Signal	Where	Meaning / action
Pause times vs target	gc.log pause lines; JFR jdk.GCPhasePause	Above target: check heap headroom, humongous allocations, collector choice
to-space exhausted / evacuation failure (G1)	gc.log	Heap too tight or allocation spikes; increase headroom, reduce allocation
Humongous allocations (G1)	gc.log with gc+humongous tags, JFR	Objects ≥ half a region; increase -XX:G1HeapRegionSize or avoid huge arrays
Allocation stalls (ZGC)	gc.log “Allocation Stall”	ZGC cannot keep up: more heap or fewer allocations
Full GC	gc.log “Pause Full”	Almost always a problem in G1: leak, explicit System.gc(), Metaspace exhaustion
GC CPU > ~10%	-Xlog:gc+cpu (user/sys/real per collection), GC thread CPU in JFR	Allocation pressure: profile allocations, then size heap
Long safepoint time to reach	safepoint log tag	Counted loops, large array copies, biased code paths; not a GC setting

jcmd <pid> JFR.start name=tune settings=profile duration=120s filename=/dumps/tune.jfr
jfr summary /dumps/tune.jfr
jfr print --events jdk.GarbageCollection /dumps/tune.jfr | head -60
jfr print --events jdk.ObjectAllocationSample --stack-depth 20 /dumps/tune.jfr | head -100

Open recordings in JDK Mission Control for allocation hot paths, lock contention, I/O and thread activity in one view. GC log visualisers (for example GCeasy-style tools) are convenient, but uploading logs to third-party sites shares production data: ask first.

Other levers worth knowing (JDK 21–25)

• Virtual threads (JDK 21+): pinning on synchronized was removed in JDK 24; on 21–23, watch jdk.VirtualThreadPinned JFR events.
• Compact object headers (-XX:+UseCompactObjectHeaders, product flag in JDK 25) cut heap use for object-heavy workloads; test before adopting.
• Startup: Class Data Sharing / AppCDS archives, and the AOT cache in JDK 24+ reduce start time; Spring Boot has built-in support for CDS training runs.
• Avoid copying old flag sets (-XX:+UseConcMarkSweepGC, PermSize, -XX:+AggressiveOpts): removed or meaningless on modern JDKs and can stop the JVM from starting.

Workflow

- [ ] 1. Write down the goal and the current numbers (pause p99, GC CPU, RSS, restarts)
- [ ] 2. Enable GC logging + JFR; capture under representative load
- [ ] 3. Rule out code problems: allocation hot spots, leaks, explicit System.gc()
- [ ] 4. Set memory sizing for the container; choose the collector for the goal
- [ ] 5. Change one thing; rerun the same load; compare
- [ ] 6. Roll out to one instance (canary) before all, with the user's approval

Verification

Same load, same duration, before vs after:

| Metric | Before | After |
|---|---|---|
| GC pause p99 | 340 ms | 2 ms |
| Request p99 | 610 ms | 220 ms |
| GC CPU | 14% | 9% |
| RSS / container limit | 1.9 / 2 GB (OOMKilled daily) | 1.5 / 2 GB |
| Throughput | 2,100 rps | 2,050 rps |

If p99 did not move when GC pauses dropped, GC was not the bottleneck; look at locks, I/O or downstream latency ($latency-budgets).

Report template

## JVM tuning: <service> (JDK <ver>, <container limits>)
Goal: request p99 < 250 ms; no OOMKilled restarts
Findings: G1 pauses up to 340 ms from humongous allocations (8 MB byte[] in report export); RSS overrun from 400 threads
Changes: -XX:+UseZGC; MaxRAMPercentage 75 -> 65; export streams instead of buffering; thread pool capped at 64
Results: <table>   Canary: <status>   Flags file: deploy/jvm.options