Argo CD GitOps skill for Codex

Git is the only way changes reach the cluster: every environment is a directory or values file, promotion is a commit, rollback is a revert, and Argo CD shows Synced and Healthy for everything because live state matches Git, not because diffs are ignored wholesale.

Inspect first

1. Read AGENTS.md, then map the GitOps repo: where Applications live, how environments are separated (directories, branches, values files), and what renders manifests (plain YAML, Kustomize, Helm).
2. Check the Argo CD version (argocd version or the server image tag). Argo CD 3.x changed defaults from 2.x, including annotation-based resource tracking and stricter RBAC inheritance; read the upgrade notes before copying 2.x examples.
3. List AppProjects and what they allow: argocd proj list, or the YAML in the repo.
4. For a broken app: argocd app get <app> and argocd app diff <app> before changing anything.

The argocd CLI needs server access and login. Offline, edit and render manifests (kustomize build, helm template) and hand the cluster steps to the user.

Repo layout

Prefer directories per environment on one branch over branch-per-environment; branches drift and merges between them carry unintended changes.

gitops/
├── bootstrap/
│   └── root-app.yaml            # app-of-apps or ApplicationSet entry point
├── projects/
│   └── shop.yaml                # AppProject
├── apps/
│   └── orders/
│       ├── base/                # Kustomize base (or a Helm chart reference)
│       └── overlays/{dev,staging,production}/
└── appsets/
    └── shop-services.yaml       # ApplicationSet: one Application per app x env

AppProject: the guardrail

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: shop
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/example/gitops.git
  destinations:
    - { server: https://kubernetes.default.svc, namespace: "shop-*" }
  clusterResourceWhitelist: []        # no cluster-scoped objects from app teams
  namespaceResourceBlacklist:
    - { group: "", kind: ResourceQuota }

Never put team apps in the default project; it allows any repo and any destination.

Application

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: orders-production
  namespace: argocd
  finalizers:
    - resources-finalizer.argocd.argoproj.io   # deleting the app deletes its resources
spec:
  project: shop
  source:
    repoURL: https://github.com/example/gitops.git
    targetRevision: main
    path: apps/orders/overlays/production
  destination:
    server: https://kubernetes.default.svc
    namespace: shop-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - CreateNamespace=true
      - ServerSideApply=true
      - PruneLast=true
    retry:
      limit: 5
      backoff: { duration: 10s, factor: 2, maxDuration: 3m }
  ignoreDifferences:
    - group: apps
      kind: Deployment
      jsonPointers: [/spec/replicas]          # owned by the HPA
  revisionHistoryLimit: 10

The resources finalizer makes deleting the Application cascade to every workload it manages. Deleting an Application is therefore a destructive production action: confirm with the user, and remove the finalizer first if the intent is only to stop managing it.

Many apps: ApplicationSet

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: shop-services
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - matrix:
        generators:
          - git:
              repoURL: https://github.com/example/gitops.git
              revision: main
              directories:
                - path: apps/*
          - list:
              elements:
                - { env: staging, namespace: shop-staging }
                - { env: production, namespace: shop-prod }
  template:
    metadata:
      name: "{{.path.basename}}-{{.env}}"
    spec:
      project: shop
      source:
        repoURL: https://github.com/example/gitops.git
        targetRevision: main
        path: "{{.path.path}}/overlays/{{.env}}"
      destination:
        server: https://kubernetes.default.svc
        namespace: "{{.namespace}}"
      syncPolicy:
        automated: { prune: true, selfHeal: true }

Pattern	Use when
Single Application	One app, one environment, hand-managed
App of apps	A small, curated set; bootstrap of platform components in a fixed order
ApplicationSet (git directories, list, cluster, matrix)	Many apps or clusters following one convention

ApplicationSets delete generated Applications when a generator stops producing them (for example a directory rename). Set spec.syncPolicy.preserveResourcesOnDeletion: true or use applicationsSync: create-update while restructuring.

Ordering: waves and hooks

metadata:
  annotations:
    argocd.argoproj.io/sync-wave: "-1"      # lower waves apply and become healthy first

Typical waves: -2 namespaces and CRDs, -1 config, secrets and migrations, 0 workloads, 1 ingress and post-deploy checks. Argo CD waits for each wave to be healthy before the next.

Database migration as a hook:

apiVersion: batch/v1
kind: Job
metadata:
  name: orders-migrate
  annotations:
    argocd.argoproj.io/hook: PreSync
    argocd.argoproj.io/hook-delete-policy: BeforeHookCreation
spec:
  backoffLimit: 1
  template:
    spec:
      restartPolicy: Never
      containers:
        - name: migrate
          image: registry.example.com/shop/orders:1.14.2
          command: ["./orders", "migrate", "up"]

Migrations must be backward compatible with the running version; a failed PreSync hook stops the sync, but schema changes already applied stay applied.

Promotion and rollback

• Promotion = a commit changing the image tag in the next environment’s overlay or values (by CI, a PR bot, or Argo CD Image Updater writing back to Git).
• Rollback = git revert of the promotion commit; Argo CD syncs the old state. This keeps Git truthful.
• argocd app rollback <app> <history-id> works only with automated sync disabled and leaves Git out of sync with the cluster. Use it as an emergency step, with user confirmation, then revert in Git and re-enable auto-sync.
• Production syncs, hard refreshes with prune, and rollbacks need the user’s explicit go-ahead.

Debugging states

State	Check	Typical cause
OutOfSync right after sync	argocd app diff	Controller or webhook mutates a field (replicas, caBundle, defaults); add a narrow ignoreDifferences plus RespectIgnoreDifferences=true
Sync failed	argocd app get <app> operation message	Schema error, AppProject denies kind or namespace, hook failed
Progressing forever	Resource health in UI, pod events	Pods not ready; debug the workload ($kubernetes-debugging)
Degraded	Health of child resources	Failed Job, CrashLoopBackOff, PVC pending
Unknown / ComparisonError	repo-server logs	Render error in Helm/Kustomize, private repo credentials, plugin failure
Manual kubectl change reverted	selfHeal	Working as designed; make the change in Git

Workflow

- [ ] 1. Map repo layout, Argo CD version, projects, existing apps
- [ ] 2. Render the target path locally (kustomize build / helm template)
- [ ] 3. Write AppProject, Application or ApplicationSet; scope the project tightly
- [ ] 4. Add sync waves and hooks only where order matters
- [ ] 5. Open a PR; after merge, watch the app sync (or argocd app sync with confirmation)
- [ ] 6. Confirm Synced + Healthy and a clean argocd app diff

Verification

kustomize build apps/orders/overlays/production | kubeconform -strict -summary -ignore-missing-schemas
argocd app get orders-production --refresh
argocd app diff orders-production            # exit code 0 means no diff
argocd app wait orders-production --health --sync --timeout 600
argocd appset get shop-services              # generated applications listed

Output

**Apps:** orders-staging, orders-production generated by ApplicationSet shop-services
**Project:** shop (repo and shop-* namespaces only, no cluster-scoped resources)
**Sync:** automated, prune + selfHeal, ServerSideApply; replicas ignored (HPA-owned)
**Order:** wave -1 migration Job (PreSync hook), wave 0 Deployment, wave 1 Ingress
**State:** Synced / Healthy in staging; production waits for the promotion PR
**Rollback:** git revert of the promotion commit (documented in README)