16.1 Deployment and Release: Rolling, Blue-Green, Canary, and Feature Flags
Suppose a team stops its service for 30 seconds on every release, and one incorrect parameter has already caused an outage lasting several hours. Deployment strategy must therefore control both exposure and rollback, not merely copy a new artifact.
Deployment moves a new artifact into production, while release is when users begin interacting with the new behavior. By separating these two steps, teams can first validate that the system runs healthily before gradually exposing business changes.
Regardless of the strategy used, the foundational principle is that the same immutable artifact progresses through environments. If each environment rebuilds independently, the binary verified in staging may not be the same as the one running in production.
First Define Verifiable Signals
A Pod in the Running phase does not guarantee that all containers are ready, much less that user operations succeed. To make a release decision, we also need:
- Technical signals: error rates, latency, saturation, restarts, and dependency failures;
- Business signals: order success rate, settlement discrepancies, registration conversion;
- Data invariants: duplicate processing, account balance or point reconciliation;
- User feedback: anomalies reported from specific client apps or tenant environments.
Release strategies are merely traffic controls. They only become safe and reliable when tied to these signals and defined stop conditions.
Rolling Updates
Rolling updates gradually replace instances, keeping low resource costs and making them well-suited for routine deployments with forward and backward compatibility.
During the coexistence of old and new versions, the following must be guaranteed:
- API and message schemas must enable communication across versions;
- Database schema changes must add compatible structures before removing old ones;
- Sessions must not depend on local state within a single instance, or must have a clear strategy for draining such state;
- Readiness probes must pass only when the instance is truly ready to receive traffic;
- Before termination, new requests must be stopped and ongoing requests must be allowed to complete.
For example, with 10 desired replicas, maxSurge: 20% allows 2 extra replicas, while maxUnavailable: 0 requires the controller to wait for new replicas to become available before reducing old ones. This needs spare schedulable capacity and does not guarantee zero business errors. Terminating Pods can temporarily push resource use beyond the budget for those 12 replicas. Exceeding progressDeadlineSeconds reports failed progress; it does not automatically roll back. The Deployment documentation explains these limits.
Rolling updates are not suitable for disruptive changes that cannot tolerate coexistence between old and new versions.
Blue-Green Deployment
Blue-Green deployment maintains two fully operational environments. After validation in the Green environment, the traffic routing is switched in a single step:
Traffic → Blue v1
Green v2 (warm-up and validation)
After switch: Traffic → Green v2
Blue v1 (rollback window)A "single switch" means one routing configuration change. Connection draining and rule propagation still take time; requests do not all change versions instantly worldwide. Queue consumers and scheduled jobs in the two environments need separate activation or ownership rules. HTTP routing does not elect their active owner.
Traffic switching and application rollback can be fast. The cost is double the resource capacity, and shared state components like databases, caches, and message queues cannot be instantly rolled back with the traffic switch.
If version v2 has written data that v1 cannot interpret, rolling back to v1 might be more dangerous than continuing with v2. Therefore, rollback capability must be proven through compatibility testing, rather than by verifying that the load balancer can switch traffic back alone.
Canary Release
Canary releases begin by sending a small, representative portion of traffic to the new version, then gradually scale up:
Internal accounts → 1% → 5% → 25% → 50% → 100%Each stage should include:
- A minimum observation window and sufficient sample size;
- Error rates and latency compared to the baseline version;
- Business metrics and key invariant values;
- Automated pause/rollback thresholds;
- A risk-based decision on whether manual approval is needed and who owns it.
These percentages are examples. If independent requests have a true failure rate of 1%, observing only 100 requests still has a 0.99^100 ≈ 36.6% chance of revealing no errors. An entirely green minute may simply contain too little evidence.
Rare errors may not be visible during low-traffic phases, and the percentage alone does not guarantee high-value tenant exposure. Traffic splitting dimensions can be tenants, regions, accounts, or request characteristics, but avoid random shuffling of the same workflow between versions.
Feature Flag: Expose Features Independently
Feature flags allow code to be deployed before exposing its behavior to everyone. This branch is a fragment of a project query method. It requires flags to provide stable cohorts and a failure default, and both query implementations to satisfy the same business contract:
if (flags.enabled("new-ranking", user.id())) {
return newRanking.query(user);
}
return legacyRanking.query(user);A feature flag must include:
- Owner, purpose, and expiration date;
- Stable user segmentation to prevent inconsistent user experiences;
- Default behavior and fallback actions when configuration services are unavailable;
- Change auditing and an emergency shutoff path;
- A scheduled deletion task after the feature is fully launched.
Turning a flag off changes subsequent routing; it does not undo new data or work already in flight. Both query implementations must keep understanding the same data throughout the rollback window.
Client-side feature flags cannot safeguard critical concerns like permission checks, financial rules, or data integrity constraints. While client-side flags control what is displayed, the server must still enforce security and business rules.
Database Changes: Expand–Migrate–Contract
Safe destructive changes typically span multiple releases:
- Expand: Add new nullable columns, tables, or compatible APIs;
- Migrate: Populate historical data to enable backward and forward compatibility between old and new code;
- Switch: Transfer read and write ownership and monitor system behavior;
- Contract: Remove old structures only after old application versions, batch jobs, and consumers have retired and the rollback window has ended.
For example, renaming player_name to display_name needs more than a historical backfill. New writes during backfill must maintain both representations or be caught up through change capture. Check that the representations agree before switching reads, then stop old writes. If one local transaction can maintain both representations, there is no need to introduce cross-database dual writes.
Large table migrations must be rate-limited, batched, and recoverable, with continuous monitoring of locks, replication lag, and storage growth. Application rollback does not imply automatic data rollback.
Choosing Deployment Strategies
| Condition | Recommended Strategy |
|---|---|
| Routine compatibility changes with limited capacity | Rolling update |
| Need for rapid, full rollout and ability to tolerate dual environments | Blue-green deployment |
| High-risk changes that can be validated incrementally by traffic | Canary deployment |
| Need to decouple deployment from user exposure | Feature Flag, used in combination with other strategies |
The next lesson explores auxiliary container patterns within the same deployment unit and how fault injection can be used to validate recovery assumptions.
References
- Kubernetes, Deployments
- Microsoft, Safe deployment practices
- Martin Fowler, Feature Toggles