15.1 Consistency Boundaries and Saga: Transforming Long Transactions into Recoverable Flows

The registration service has reserved a seat, but the payment service has not replied in time. Each database has committed its local transaction successfully, yet the user still does not know whether they can enter the tournament.

After services take ownership of their own data, business workflows across services can no longer rely on a single local database transaction. The design must define which invariants must hold immediately, which states can temporarily be inconsistent, and how the system should recover in the event of failure.

Write Invariants First, Then Choose Patterns

Take the event registration flow as an example:

• The number of confirmed spots for a single match cannot exceed its capacity.
• A player cannot be charged twice for the same registration payment.
• If payment succeeds but registration fails, a refund must be issued or the case must be escalated to manual handling.
• The leaderboard may allow a one-minute delay in updates.

The first two conditions may require strong constraints, refund or manual handling needs reliable recovery, and the last permits bounded lag. Eventual consistency itself promises no one-minute deadline; that business target also needs lag metrics, alerts, and repair. If these are not clearly distinguished, the team will end up enforcing expensive strong consistency across all data, or (more dangerously) will incorrectly accept "it will eventually work out" as a valid state for critical financial operations.

Where possible, place invariants that must hold atomically within the same data owner. If a service boundary cuts across a core invariant, no matter how sophisticated the messaging pattern becomes, it can only patch over a fundamentally flawed boundary design.

A Saga Is a Sequence of Local Transactions

Saga breaks long-running workflows into multiple local transactions committed by different services. A confirmed business failure triggers compensation for committed results. Transient failures can be retried first. Unknown outcomes need a query or confirmation; a timeout is not proof that a step never ran.

Normal: reserve seat → authorize payment → confirm registration
Authorization definitively fails: release seat
Confirmation definitively fails: void authorization → release seat

This example assumes registration only authorizes the payment. Authorization has not captured funds; capture and recovery from its failures need a separately defined step. If funds have already been captured, compensation generally requires a refund, not voiding an authorization.

Compensation is not database rollback. Refunds may incur fees, notifications cannot be "unremembered," and external systems might have already processed the changes. Therefore, compensation must have clear business semantics and allow for manual intervention.

Orchestration and Choreography

Orchestration

The orchestrator explicitly records workflow state and sends commands to proceed to the next step:

Registration Saga
  ├─ ReserveSeat
  ├─ AuthorizePayment
  ├─ ConfirmRegistration
  └─ on failure: ReleaseSeat / VoidPayment

The advantage is that state, timeouts, and compensation are all centralized and visible. The risk is that the orchestrator absorbs too many domain-specific decisions, becoming a monolithic "god object" spanning multiple services.

Choreography

Services subscribe to events and react independently:

SeatReserved → Payment authorizes
PaymentAuthorized → Registration confirms
RegistrationConfirmed → Notification sends

The benefit is that publishers do not need to know about all consumers. The downside is that the full workflow is scattered across multiple subscribers, making it harder to detect circular dependencies, duplicate processing, or failure points.

Use choreography when the workflow is simple, stable, and each subsequent action is independent. Use orchestration when the process involves many steps, complex compensation logic, or requires clear SLAs and manual intervention.

Intermediate States Must Enter the Model

Long-running workflows can't be represented by just "success" or "failure." Enrollment might be in one of these states:

PENDING_SEAT
PENDING_PAYMENT
CONFIRMED
CANCELLING
CANCELLED
MANUAL_REVIEW

A Saga provides no transaction isolation across services; other workflows can observe committed intermediate results. These state names are examples, not a complete transition graph. Define the race between seat expiry and a late payment result. If the seat has already been released when payment succeeds, enter void or refund recovery instead of marking the registration confirmed.

These states provide a basis for retries, queries, and manual intervention. Semantic locks or state machines prevent conflicting operations, for example, PENDING_PAYMENT prevents re-initiating a payment on the same transaction.

This conditional update is a project fragment. registration.id must be unique, and :name denotes client-bound parameters, not variables that can be pasted directly into psql. The caller must already have verified payment credentials, amount, and a valid seat reservation. The state and version predicates only prevent concurrent overwrites:

UPDATE registration
SET status = 'CONFIRMED', version = version + 1
WHERE id = :id
  AND status = 'PENDING_PAYMENT'
  AND version = :expected_version;

A zero-row count only means no row matched this state and version. The ID may also be absent or the row invisible to the caller. Query again and decide whether to return idempotently, recover, or reject; do not classify every zero count as a retryable version conflict.

Every Step Must Define Its Recovery Semantics

Problem	Must Be Clearly Defined
Command Reexecution	Is it idempotent? How long should the idempotency key be stored?
Response Timeout	How should the result be queried, rather than blindly retrying?
Event Out-of-Order	Should we reject based on version, buffer, or reorder events?
Compensation Failure	Should we retry? When should human intervention be triggered?
Flow Timeout	What is the business deadline, and what is the automatic cancellation strategy?
Service Restart	From where should the workflow state be restored?

The reliability of a Saga comes from persistent state, idempotent execution, and observable recovery, not from a smooth, successful happy path.

The next lesson addresses the atomic gap between local transactions and message publishing, and establishes consumer idempotency and dead-letter handling.

References

• Chris Richardson, Saga
• AWS Prescriptive Guidance, Saga orchestration pattern