10.2 GitOps: Continuous Synchronization, Deployment, and Rollback

Traditional pipelines often execute the final step (pushing a change into production) after acquiring cluster credentials in CI. GitOps reverses this flow: cluster controllers continuously read the approved desired state, compare it against the current live state, and reconcile any differences back to the desired configuration.

CI: source code → testing → build immutable artifacts → sign/scanning
CD: desired state repository or OCI artifact → coordinator → cluster

GitOps isn't simply "put all YAML files into Git." The core lies in declarative state, versioned sources, automatic pull mechanisms, and continuous reconciliation, forming a closed-loop control cycle.

kubectl apply

Git Doesn't Have to Store Source Code Directly

A traceable release chain must maintain a fixed, immutable identity:

source commit
  → image: registry.example/atlas@sha256:...
  → deployment manifest commit / signed OCI artifact
  → reconciler observed revision
  → running Pod imageID

Environment configurations should use image digests, or automated systems should update verified digests into the registry. Writing only :latest would cause the same Git commit to resolve to different binaries at different times, distorting rollbacks and audits.

Configuration sources can be Git, or supported OCI artifacts or Helm repositories. What matters isn't the source name, but that the source is verifiable, versions are unambiguous, and changes are approvable.

Coordination Loops Are Not a One-Time Deployment

Take Argo CD as an example: an Application specification defines its source, target, and synchronization strategy:

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: atlas-api
  namespace: argocd
spec:
  project: production
  source:
    repoURL: https://git.example.com/platform/env-config.git
    targetRevision: 7f3c91e
    path: clusters/prod/atlas-api
  destination:
    server: https://kubernetes.default.svc
    namespace: atlas-prod
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
    syncOptions:
      - PruneLast=true

selfHeal enables the controller to fix manual drifts in the live state; prune allows deletion of objects that no longer exist in the source. Both alter failure modes: erroneous commits can rapidly propagate across the entire cluster, so production environments must enforce project boundaries, synchronization windows, health checks, and confirmation prompts for critical resource deletions.

When automatic synchronization is enabled, do not treat "clicking back to an older version" as a long-term solution: if Git still declares a new version, the coordinator will redeploy it. A reliable rollback should generate a new, auditable desired state change, such as reverting a configuration commit or rolling back to a known-good digest version.

Repository Boundary Mapping of Permissions and Release Cadence

A common layout:

env-config/
├── apps/
│   └── atlas-api/base/
└── clusters/
    ├── staging/atlas-api/
    └── prod/atlas-api/

Application teams manage the base configuration and app-specific parameters, while platform teams handle cluster capabilities and policies. Changes to production environments require independent CODEOWNERS approval. Avoid copying entire YAML files to create three progressively divergent environments; instead, use explicit mechanisms like Kustomize overlays, Helm values, or other well-defined configuration combinations to express only the necessary differences.

Repositories should also not be so large that every commit triggers a full recalculation across all clusters. It's more important to partition repositories based on permissions, failure impact, and change frequency than to pursue a purely monorepo or multi-repo structure for formality's sake.

Promotion is an upgrade of the same artifact

Correct environment promotion does not involve building the image digest twice, one for staging and one for production:

Build once, image digest D
  → staging references D
  → integration, performance, and strategy validation
  → PR updates production reference to the same D
  → phased rollout and business validation

This ensures production issues can be traced back to a single, previously tested artifact. Environment-specific configurations must be decoupled from the artifact identity. Database migrations must explicitly enforce forward compatibility, define execution order, and include failure recovery mechanisms, assumptions that rolling back the image will automatically restore data are unsafe and invalid.

Secrets Should Not Be Entered into Expected State in Plain Text

Git can store declarations of "which secrets are needed," but it should never contain the secrets themselves. Common approaches include:

• External Secrets controllers that pull secrets from cloud-based Secret Managers or HashiCorp Vault;
• SOPS-encrypted manifests, where keys are stored in KMS and decryption occurs only within controlled environments;
• CSI drivers that mount secrets directly into workloads, reducing the need to replicate Kubernetes Secrets across pods.

Base64 is merely encoding. Even when using encrypted files, decryption must be restricted to authorized identities, keys must be rotated regularly, and secrets must never appear in diffs, logs, or rendered artifacts.

Break-glass Must Include a Return Path

In an incident, it may be necessary to pause coordination or directly patch the live state. Mature processes predefine:

1. Who can pause which application and how long credentials remain valid;
2. When work orders, commands, and impact scopes are recorded;
3. How effective fixes are submitted back to the expected state after service recovery;
4. How coordination is restored and verified to ensure no secondary rollbacks occur.

Without a return path, kubectl edit becomes a seed for the next incident. Completely banning emergency actions would slow recovery. The goal of GitOps is to make exceptions controllable, temporary, and traceable.

Pre-Deployment Control Plane Verification

• Is the source fixed to a traceable revision or digest?
• Does the coordinator allow writes only to authorized namespaces and resource types?
• Do deletion confirmations, sync windows, and risk assessments align for prune, selfHeal, and other operations?
• Are there ordering constraints between CRDs and their instances, database migrations, and application versions?
• If a controller fails, does the current workload continue to run?
• Can we map a Git revision to the image ID currently in use?
• Have emergency pause, rollback, and recovery coordination procedures been exercised?

Infrastructure-as-Code and GitOps both use declarative configurations, but they differ in ownership: the former typically manages cloud resources and the cluster foundation, while the latter continuously enforces the desired state within the cluster. When both controllers attempt to modify the same field simultaneously, the result is not redundancy, but an endless cycle of conflict and re-convergence.

References

• OpenGitOps, GitOps Principles
• Argo CD, Automated Sync Policy
• Argo CD, Sync Options
• Flux, Core Concepts