Service Mesh: Istio and Linkerd skill for Codex

Done well means: the mesh solves a named problem (mTLS everywhere, L7 traffic control, uniform telemetry) that simpler tools could not, it was rolled out namespace by namespace in permissive mode first, and the team can explain any 503 using istioctl or linkerd output rather than guesswork.

Decide first: do you need a mesh?

Need	Without a mesh	Mesh worth it?
Encrypt service-to-service traffic	Cilium/Calico WireGuard node encryption, app-level TLS	Only if you also need workload identity and per-service authz
Canary by percentage	Argo Rollouts or Flagger with ingress-nginx/Gateway API, $deployment-strategies	Only for east-west (service-to-service) canaries
Retries, timeouts	Client libraries, gRPC service config	If many languages make library-level consistency impractical
Golden-signal metrics per edge	OpenTelemetry instrumentation	Mesh gives it without code changes
Zero-trust authz between services (who may call whom)	NetworkPolicy (L3/L4 only)	Yes, with identity-based L7 policy

If the honest answer is “fewer than ~10 services and one language”, recommend not adopting a mesh and say what to use instead. A mesh adds a control plane to upgrade, new failure modes on every request path and a learning curve for on-call.

Choose the mesh

Option	Pick when	Costs
Istio ambient (ztunnel L4 + optional waypoint L7)	New adoption on Istio; want mTLS without sidecars or pod restarts	Newer mode; some sidecar features need a waypoint or are not yet supported, check docs for your version
Istio sidecar	Need mature per-pod L7 features, multicluster setups already proven on sidecars	Envoy per pod (CPU, memory), injection restarts, Job/startup ordering issues
Linkerd	Want the simplest mesh with mTLS on by default and low overhead	Fewer L7 features; stable release builds come from Buoyant with licensing terms for larger companies, while the open-source project ships edge releases

Check the cluster version and CNI first (kubectl version, kubectl get ds -n kube-system). Ambient needs a supported CNI configuration; managed offerings (GKE, AKS Istio add-on, EKS with Istio) have their own install paths. Read AGENTS.md and the GitOps repo: the mesh should be installed the way everything else is (Helm via Argo CD, not ad-hoc istioctl install on production).

Rollout workflow

- [ ] 1. Install control plane in a non-prod cluster; run the mesh's check command
- [ ] 2. Enrol one low-risk namespace; mTLS PERMISSIVE
- [ ] 3. Confirm traffic, metrics and probes still work; compare error rates
- [ ] 4. Enrol remaining namespaces one by one
- [ ] 5. Switch to STRICT mTLS per namespace, then mesh-wide
- [ ] 6. Add AuthorizationPolicy (deny-by-default per namespace) once callers are known
- [ ] 7. Only then add traffic shifting, retries and timeouts

Enrolling or switching production namespaces changes every request path: get the user’s confirmation before each production step.

Istio essentials

istioctl x precheck
istioctl install --set profile=ambient --skip-confirmation      # non-prod; Helm for prod
kubectl label namespace shop istio.io/dataplane-mode=ambient     # ambient enrolment
# Sidecar mode instead (restart pods afterwards):
kubectl label namespace shop istio-injection=enabled
kubectl rollout restart deploy -n shop
istioctl analyze -A

mTLS, permissive first, strict later:

apiVersion: security.istio.io/v1
kind: PeerAuthentication
metadata:
  name: default
  namespace: shop          # put it in istio-system for mesh-wide
spec:
  mtls:
    mode: STRICT           # start with PERMISSIVE

Identity-based authorization (allow only the frontend’s service account to call cart):

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: cart-allow-frontend
  namespace: shop
spec:
  selector:
    matchLabels: { app: cart }
  action: ALLOW
  rules:
    - from:
        - source:
            principals: ["cluster.local/ns/shop/sa/frontend"]

Once any ALLOW policy selects a workload, everything not matched is denied. In ambient mode, L7 rules (paths, methods) need a waypoint for that namespace or service (istioctl waypoint apply -n shop --enroll-namespace); ztunnel enforces L4 only.

Traffic shifting

Prefer Gateway API (works in ambient and sidecar mode, and with Linkerd):

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: cart
  namespace: shop
spec:
  parentRefs:
    - group: ""
      kind: Service
      name: cart
  rules:
    - backendRefs:
        - name: cart-v1
          port: 8080
          weight: 90
        - name: cart-v2
          port: 8080
          weight: 10
      timeouts:
        request: 5s

Classic Istio sidecar API: a DestinationRule defining subsets by version label plus a VirtualService with weighted route entries. Do not mix VirtualService and HTTPRoute for the same host.

Move weights in steps (1, 10, 50, 100) gated on error rate and latency from the mesh metrics. Automate with Flagger or Argo Rollouts rather than hand-editing weights.

Linkerd essentials

linkerd check --pre
linkerd install --crds | kubectl apply -f -
linkerd install | kubectl apply -f -
linkerd check
kubectl annotate namespace shop linkerd.io/inject=enabled
kubectl rollout restart deploy -n shop
linkerd viz install | kubectl apply -f - && linkerd viz stat deploy -n shop

mTLS is automatic between meshed pods. Use Server and AuthorizationPolicy (policy.linkerd.io) for who-may-call-whom, and Gateway API HTTPRoute weights for traffic splitting.

Debugging

istioctl proxy-status                                  # is every proxy SYNCED with istiod?
istioctl x describe pod <pod> -n shop                  # policies and routes that apply
istioctl proxy-config routes deploy/frontend -n shop   # what Envoy actually has
istioctl proxy-config clusters deploy/frontend -n shop | grep cart
kubectl logs <pod> -c istio-proxy -n shop | tail       # sidecar access log / errors
kubectl logs -n istio-system ds/ztunnel | grep shop    # ambient L4 path
linkerd viz tap deploy/frontend -n shop                # live requests (Linkerd)

Symptom	Likely cause
503 UC / upstream connect error after STRICT	A caller is not in the mesh and still sends plaintext
503 NR (no route)	VirtualService/HTTPRoute host or port mismatch; run istioctl analyze
HTTP features ignored, traffic treated as TCP	Port not named http-*/grpc-* and no appProtocol on the Service
App fails at startup, cannot reach network	Container started before the proxy; set holdApplicationUntilProxyStarts: true or use native sidecars
Jobs never complete	Sidecar keeps running; use native sidecars (Kubernetes 1.29+, check istiod for ENABLE_NATIVE_SIDECARS) or ambient
Liveness probes fail after injection	Probe rewriting disabled or app binds to localhost only
Headless service / StatefulSet peers fail	Pod-to-pod addressing needs explicit ServiceEntry or port naming checks
Latency jumps after enabling retries	Retries on non-idempotent calls or retry storms; set per-try timeouts and budgets

Upgrades

• Istio: canary upgrades with revisions (istio.io/rev=1-xx labels, or revision tags), move namespaces gradually, keep the old control plane until no proxy reports it in istioctl proxy-status. Check the release notes for how many minor versions an upgrade may jump; in-place upgrades are one minor version at a time.
• Linkerd: upgrade the control plane, then restart workloads so proxies pick up the new version; linkerd check --proxy confirms versions match.

Verification

• istioctl analyze -A (or linkerd check) reports no errors.
• mTLS is real: istioctl x describe pod shows STRICT, or linkerd viz edges deploy -n shop shows secured edges; a plaintext request from an unmeshed debug pod is rejected.
• Error rate and p99 latency per service match the pre-mesh baseline within noise.
• A canary weight change is visible in per-version request metrics.

Report template

**Decision:** Istio ambient (reason: mTLS + identity authz across 40 services, 4 languages)
**Rolled out:** staging all namespaces; prod: payments, cart (PERMISSIVE)
**Policies:** STRICT in staging; cart accepts only sa/frontend and sa/checkout
**Checks:** istioctl analyze clean; p99 +1.8 ms; 5xx unchanged; plaintext probe rejected
**Next (needs approval):** STRICT in prod payments namespace on Tuesday window

Kubernetes-wide controls (RBAC, pod security, NetworkPolicy) live in $kubernetes-security.