10.2 GitOps: Continuous Reconciliation, Deployment, and Rollback

Traditional pipelines often end with kubectl apply: CI obtains cluster credentials and pushes a change into production. GitOps reverses this flow: trusted reconcilers continuously read the approved desired state, compare it against the current live state, and reconcile any differences back to the desired configuration.

CI: source code → testing → build immutable artifacts → sign/scan
CD: desired state repository or OCI artifact → reconciler → cluster

GitOps isn't simply "put all YAML files into Git." The core lies in declarative state, versioned sources, automatic pull mechanisms, and continuous reconciliation, forming a closed-loop control cycle.

Desired State Needs a Traceable Source

A traceable release chain must maintain a fixed, immutable identity:

source commit
  → image: registry.example/atlas@sha256:...
  → deployment manifest commit / signed OCI artifact
  → reconciler observed revision
  → running Pod imageID

Environment configurations should use image digests, or automated systems should update verified digests in the repository. Writing only :latest would cause the same Git commit to resolve to different binaries at different times, distorting rollbacks and audits.

The digest in the diagram is a placeholder; deployments need the full value. A multi-platform image index and the platform image running on a node can have different digests. Preserve that resolution chain instead of comparing two strings alone. Sources can also be OCI artifacts or Helm repositories supported by the controller; when pinning versions, check whether the repository permits replacing content under the same version.

Reconciliation Continues After Deployment

An Argo CD Application defines its source, target, and sync policy. This example requires an installed Argo CD, an existing production AppProject and atlas-prod namespace, and project rules and repository credentials that permit the source and destination. Replace the repository URL and commit placeholder:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: atlas-api
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://git.example.com/platform/env-config.git
    targetRevision: "<approved-full-commit-sha>"
    path: clusters/prod/atlas-api
  destination:
    server: https://kubernetes.default.svc
    namespace: atlas-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - PruneLast=true

selfHeal repairs manual drift in managed objects; prune removes managed objects deleted from the source. Bad commits can spread within the authorized scope, so restrict project destinations and configure sync windows and health checks. PruneLast=true delays pruning until other sync waves succeed and resources are healthy; it does not require deletion approval. Critical objects can separately use Prune=confirm.

Pinning targetRevision to a commit prevents following later branch commits. Promotion must update that reference through the process that manages the Application; committing new configuration alone leaves the application pinned.

Argo CD does not permit a direct historical rollback while automated sync is enabled. Normally, revert the configuration or restore a known-good digest in a new desired-state commit. With commit pinning, also update targetRevision to that commit. When tracking a protected branch, the new commit is discovered as the branch advances.

Repository Boundaries Follow Permissions and Release Cadence

A common layout:

env-config/
├── apps/
│   └── atlas-api/base/
└── clusters/
    ├── staging/atlas-api/
    └── prod/atlas-api/

Application teams manage the base configuration and app-specific parameters, while platform teams handle cluster capabilities and policies. Production changes require approval from the relevant code owners. A CODEOWNERS file alone does not enforce approval; branch protection or repository rules must require it. Avoid copying entire YAML files to create three progressively divergent environments; instead, use explicit mechanisms like Kustomize overlays, Helm values, or other well-defined configuration combinations to express only the necessary differences.

Repositories should also not be so large that every commit triggers a full recalculation across all clusters. It's more important to partition repositories based on permissions, failure impact, and change frequency than to pursue a purely monorepo or multi-repo structure for formality's sake.

Promotion Moves the Same Artifact Between Environments

Environment promotion reuses one build for staging and production:

Build once, image digest D
  → staging references D
  → integration, performance, and policy validation
  → PR updates production reference to the same D
  → phased rollout and business validation

This ensures production issues can be traced back to a single, previously tested artifact. Environment-specific configurations must be decoupled from the artifact identity. Database migrations must explicitly enforce forward compatibility, define execution order, and include failure recovery mechanisms. Rolling back an image does not automatically restore data.

Keep Plaintext Secrets Out of Desired State

Git can store declarations of "which secrets are needed," but it should not contain plaintext secrets. Common approaches include:

• External Secrets controllers that pull secrets from cloud-based Secret Managers or HashiCorp Vault;
• SOPS-encrypted manifests, with keys in KMS and a supported decryption integration configured for the reconciler;
• Secrets Store CSI Driver mounting values from external storage; without its optional synchronization feature, this can avoid creating Kubernetes Secret objects.

Base64 is merely encoding. Even when using encrypted files, decryption must be restricted to authorized identities, keys must be rotated regularly, and secrets must never appear in diffs, logs, or rendered artifacts.

Break-glass Must Include a Return Path

An incident may require pausing reconciliation or patching live state. First check whether an ApplicationSet or parent application manages the Application: it may undo a manual change to automated sync. The process should define:

1. Who can pause which application and how long credentials remain valid;
2. When work orders, commands, and impact scopes are recorded;
3. How effective fixes are committed back to desired state after service recovery;
4. How reconciliation is restored and verified to ensure no secondary rollbacks occur.

Without a return path, kubectl edit becomes a seed for the next incident. Completely banning emergency actions would slow recovery. The goal of GitOps is to make exceptions controllable, temporary, and traceable.

Pre-Deployment Control Plane Verification

• Is the source fixed to a traceable revision or digest?
• Does the reconciler allow writes only to authorized namespaces and resource types?
• Do deletion confirmations, sync windows, and risk assessments align for prune, selfHeal, and other operations?
• Are there ordering constraints between CRDs and their instances, database migrations, and application versions?
• If a controller fails, does the current workload continue to run?
• Can we map a Git revision to the image ID currently in use?
• Have emergency pause, rollback, and resumption of reconciliation procedures been exercised?

Infrastructure-as-Code and GitOps both use declarative configurations, but they differ in ownership: the former typically manages cloud resources and the cluster foundation, while the latter continuously enforces the desired state within the cluster. When both controllers attempt to modify the same field simultaneously, the result is not redundancy, but an endless cycle of conflict and re-convergence.

References

• OpenGitOps, GitOps Principles

• Argo CD, Automated Sync Policy

• Argo CD, Sync Options

• Flux, Core Concepts

• Argo CD, Tracking strategies

• GitHub, About code owners

• Flux, SOPS integration

• Secrets Store CSI Driver, Optional Kubernetes Secret synchronization