16.2 Sidecar, Ambassador, Adapter, and Fault Injection Experiments

The sidecar container pattern moves capabilities closely tied to the main application's lifecycle but not part of its core business logic into the same Pod. They share network and some storage, as well as scaling, scheduling, and failure handling.

This coupling is sometimes exactly the goal, but it's not a reason to shove all platform features into the sidecar.

Sidecar: Auxiliary capabilities that run alongside the main container

A typical Sidecar includes log forwarding, proxies, configuration refresh, or local synchronizers:

apiVersion: v1
kind: Pod
spec:
  containers:
    - name: app
      image: registry.example/scoring@sha256:...
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
    - name: log-forwarder
      image: registry.example/log-forwarder@sha256:...
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
  volumes:
    - name: logs
      emptyDir: {}

When evaluating, calculate each Pod's additional CPU, memory, startup time, and attack surface. 1,000 application replicas mean 1,000 Sidecars, not a shared proxy.

Ambassador: Handles external communication for the representative application

Ambassador is a local proxy for external dependencies, responsible for TLS, service discovery, connection pooling, and some retry functionality:

Application → localhost proxy → External Service

The application obtains a stable local endpoint, but proxy configurations must align with business semantics. If the proxy automatically retries non-idempotent writes, the main application might never be aware that the write has occurred multiple times.

The service mesh data plane is typically implemented using Sidecar or node proxies to perform similar functions, but it cannot replace business-level deadlines, idempotency keys, or fallback policies.

Adapter: Standardize the Main Application Output

Adapter containers convert application-specific formats into a platform-wide standard, such as converting custom metrics into Prometheus exposition format or transforming legacy logs into structured events.

If you can directly modify the app to use standard protocols, it's usually simpler in the long run. Adapters are better suited for legacy systems, third-party images, or migration phases.

Init Container vs. Sidecar

One-time pre-startup tasks, like pulling static configurations, generating certificate files, or performing permission setup, are well-suited for Init Containers. Continuous capabilities that run alongside the application are better suited for Sidecar containers.

Database schema migrations should not have every replica's Init Container competing to execute simultaneously. They should be locked, audited, observable, and capable of independent rollout and rollback.

Design health checks and termination behavior as a whole

Auxiliary containers introduce new lifecycle issues:

• The main application is ready, but the proxy is not yet ready;
• After the main application exits, the log forwarder hasn't been drained yet;
• The Sidecar crashes but the Pod is still considered ready;
• Grid proxies prevent or delay application graceful termination.

Readiness, startup order, termination grace period, and drain procedures must be validated in a real deployment environment.

Chaos engineering starts with assumptions

Chaos engineering isn't about randomly breaking production; it's about using controlled experiments to test the assumption of system stability:

When a single scoring instance is terminated, settlement success remains above 99.9%, and the p99 latency recovers to baseline within 2 minutes.

An experiment includes:

1. Define an observable steady state;
2. Propose specific fault hypotheses;
3. Choose the smallest explosion radius;
4. Set auto-termination conditions;
5. Injection instance termination, network latency, dependency errors, or resource pressure;
6. Document findings and fixes, then repeat the experiment.

First validate tools and safeguards in a test environment, then gradually expand into production with small, non-critical traffic. Do not scale the experiment without monitoring, rollback capabilities, and an incident response owner.

The Real Problem with Verification

• Is the proxy timeout less than the caller's deadline?
• Are retries stacked across multiple levels?
• How would sidecar resource exhaustion affect the main container?
• Are in-transit requests and messages safely handled when an instance is terminated?
• Is the system rapidly rejecting or unbounded queuing when a region or dependency fails?
• Did the alert trigger before or at the same time as user impact?

References

• Kubernetes, Pods
• Kubernetes, Sidecar Containers
• Principles of Chaos Engineering, Principles