16.2 Sidecar, Ambassador, Adapter, and Fault Injection Experiments

The sidecar container pattern moves capabilities closely tied to the main application's lifecycle but not part of its core business logic into the same Pod. Containers in one Pod share networking; files are shared only through jointly mounted volumes. They are scheduled and scaled as one Pod. A container can restart independently, while Pod eviction or a node failure affects the group.

This coupling is sometimes exactly the goal, but it's not a reason to shove all platform features into the sidecar.

Sidecar: Auxiliary capabilities that run alongside the main container

Typical sidecars forward logs, proxy requests, refresh configuration, or synchronize local data. This example uses native sidecars, stable since Kubernetes 1.33: an initContainers entry with container-level restartPolicy: Always. It is a configuration illustration, not a deployable manifest: both image names and digests are placeholders. Real images must agree on log paths, permissions, and forwarding destinations, with resource and probe configuration added:

apiVersion: v1
kind: Pod
metadata:
  name: scoring-example
spec:
  initContainers:
    - name: log-forwarder
      restartPolicy: Always
      image: "registry.example/log-forwarder@sha256:<forwarder-digest>"
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
          readOnly: true
  containers:
    - name: app
      image: "registry.example/scoring@sha256:<app-digest>"
      volumeMounts:
        - name: logs
          mountPath: /var/log/app
  volumes:
    - name: logs
      emptyDir: {}

An emptyDir survives container restarts within the same Pod, but its logs do not survive Pod deletion. It is not a durable log backup.

Include each Pod's extra CPU, memory, startup time, and attack surface in the calculation. With 1,000 replicas and an assumed extra 50 MiB per sidecar, memory use totals about 48.8 GiB. A small proxy still sends a resource bill.

Ambassador: Handle External Communication on the Application's Behalf

Ambassador is a local proxy for external dependencies, responsible for TLS, service discovery, connection pooling, and some retry functionality:

Application → localhost proxy → External Service

The application obtains a stable local endpoint, but proxy configurations must align with business semantics. If the proxy automatically retries non-idempotent writes, the main application might never be aware that the write has occurred multiple times.

The service mesh data plane is typically implemented using Sidecar or node proxies to perform similar functions, but it cannot replace business-level deadlines, idempotency keys, or fallback policies.

Adapter: Standardize the Main Application Output

Adapter containers convert application-specific formats into a platform-wide standard, such as converting custom metrics into Prometheus exposition format or transforming legacy logs into structured events.

If you can directly modify the app to use standard protocols, it's usually simpler in the long run. Adapters are better suited for legacy systems, third-party images, or migration phases.

Regular Init Containers and Native Sidecars

A regular init container must finish successfully before initialization and the application proceed. A native sidecar keeps running after it starts. If the application requires a working proxy, give that proxy a startupProbe; a running process does not prove an established connection. An auxiliary container directly under containers remains a valid traditional multi-container pattern, but does not automatically receive these startup and shutdown ordering guarantees.

Database schema migrations should not have every replica's Init Container competing to execute simultaneously. Make migration a separately observable, locked, and audited release step with a defined compatible rollback or forward repair plan. There is no universal undo button after dropping a column.

Design health checks and termination behavior as a whole

Auxiliary containers introduce new lifecycle issues:

• The main application is ready, but the proxy is not yet ready;
• After the main application exits, the log forwarder hasn't been drained yet;
• The Sidecar crashes but the Pod is still considered ready;
• Mesh proxies prevent or delay application graceful termination.

A native sidecar readiness probe contributes to Pod readiness. During termination, the kubelet waits for the main containers to stop, then terminates native sidecars in reverse declaration order. They share the Pod grace period; if the application consumes it, the log forwarder may have no time left to drain. The native sidecar documentation describes this ordering. Readiness, probes, grace periods, and draining still need validation in the real environment.

Chaos engineering starts with assumptions

Chaos engineering isn't about randomly breaking production; it's about using controlled experiments to test the assumption of system stability:

Example hypothesis: after one scoring instance is terminated, success over logical settlement requests in the experiment window remains above 99.9%, and p99 latency returns to the agreed baseline range within 2 minutes.

This is not a measured result. Fix the traffic, observation window, and baseline tolerance before testing. Combine retries into the original logical request and refuse to declare success from an insufficient sample.

An experiment includes:

1. Define an observable steady state;
2. Propose specific fault hypotheses;
3. Choose the smallest blast radius;
4. Set automatic abort conditions;
5. Inject instance termination, network latency, dependency errors, or resource pressure;
6. Record findings and fixes, then retest the original hypothesis.

For example, restrict a latency injection to one test tenant, one instance, and at most two minutes. Stop further injection if settlement failures exceed the preset threshold, then observe recovery. Aborting the injection cannot recover lost requests or roll back business data; recovery steps must be prepared too.

First validate tools and safeguards in a test environment, then gradually expand into production with small, non-critical traffic. Do not scale the experiment without monitoring, rollback capabilities, and an incident response owner.

Questions to Test

• Is the proxy timeout less than the caller's deadline?
• Are retries stacked across multiple levels?
• How would sidecar resource exhaustion affect the main container?
• Are in-flight requests and messages safely handled when an instance is terminated?
• Is the system rapidly rejecting or unbounded queuing when a region or dependency fails?
• Did the alert trigger before or at the same time as user impact?

References

• Kubernetes, Pods
• Kubernetes, Volumes: emptyDir
• Kubernetes, Sidecar Containers
• Principles of Chaos Engineering, Principles