The distributed transaction is the question every microservices architecture meets in its second month, and the industry's answer history is a story of retreat. The two phase commit protocol, XA, promised transactional integrity across databases, the coordinator asks every participant to prepare, then commits them all, and the promise held together an era of application servers and homogeneous databases. The modern architecture that came after, heterogeneous stores, queues, cloud services, networks that partition, found the price too high, and the patterns that replaced the protocol are the Saga and the Outbox, consistency as an application level choreography rather than a coordinator's lock step. This article builds the modern patterns properly, why XA lost, how the saga composes local transactions, how the outbox makes the first step atomic, and the failure modes the patterns trade in for the locks they traded away.
Why XA retreated and what it promised while it stood
The two phase commit's problem statement is legitimate, two databases, one logical operation, both commit or neither, and the protocol delivers it through a coordinator that holds every participant in a prepared state, locks held, until the global decision. The retreat was not about correctness, it was about the operational arithmetic. The prepared state holds locks across every participant for the coordinator's round trips, so one slow participant slows every transaction it touches. The coordinator is a single failure domain, a coordinator crash leaves participants locked until recovery, and the recovery procedures are the folklore every XA operator can recite. The breadth of support narrowed, queues and caches and cloud services never joined the protocol, so every real architecture had a boundary XA could not cross anyway.
The honest assessment keeps both halves. XA works, it is implemented in the major relational engines, and a small homogeneous deployment with a proven coordinator can run it for years, the pattern's remaining audience. What it cannot do is compose the modern system's actual surface, the queue publish that must happen with the database write, the third party call that has no prepare phase, the service written last year by a team that never installed the resource adapter, and the distributed transaction at that boundary is not a protocol away, it is an architecture away.
The saga as a choreography of local transactions
The saga replaces the global transaction with a sequence of local ones, each atomic in its own database, and the integrity is composed by the sequence, complete the chain and the operation is done, and compensate a completed step when a later step fails. The order flow is the canonical chain, create the order, reserve the stock, charge the payment, each a local transaction in its owning service, and the failure of the third step triggers the compensations, release the reservation, mark the order cancelled, each compensation itself a local transaction that undoes its step. The invariant XA held by locks, no intermediate state is visible, is surrendered on purpose, intermediate states exist and are the system's real history, and the business decides which ones it can tolerate, the order that exists for a second before payment confirmation, the stock that is reserved and then released.
The composition comes in two dialects, and the choice is an organizational decision. Choreography is the event driven version, each service reacts to the previous event and emits its own, and the chain lives in no single codebase, visible only in the event flow, which suits teams that own their slices and suffer central coordinators. Orchestration is the explicit version, a saga component invokes each step and its compensation, the chain is a readable state machine in one place, and the debugging, the versioning and the onboarding all favor it at the cost of the central component that the choreography dialect avoids. The production pattern is the same one the event driven and Kafka discussions keep reaching, small teams with clear ownership can choreograph, and flows that span business departments want an orchestrator everyone can read.
The outbox as the atomic bridge out of the database
The saga's weakest link is its first hop, the local transaction that must also publish the event, and the naive implementation writes the database and publishes to the queue in two steps that can fail between, the committed order with no event, the event with no order. The outbox pattern closes the gap with the observation that the event is data, and data belongs in the database:
BEGIN;
UPDATE orders SET status = 'created' WHERE id = $1;
INSERT INTO outbox (aggregate_id, event_type, payload)
VALUES ($1, 'order.created', '{"sku": "SKU-9"}');
COMMIT;The event is committed in the same local transaction as the state change, atomic by the database's own guarantee, and a relay outside the transaction reads the outbox and publishes to the broker, marking rows as sent. The relay is either the team's own poller, a small loop with the same discipline the backup article's checks used, or the change data capture machinery the modern platforms ship, the binlog and WAL readers that turn the outbox table into a stream without a poller. The consumer side completes the pattern with idempotency, the relay retries on failure so delivery is at least once, and the receiving service must treat the duplicate event as a no operation, the dedup by event identifier that every consumer in this series carries.
The outbox costs one table, one relay and one idempotent consumer, and it buys the exact property XA bought for the queue case, the publish and the commit are one fact, plus the property XA never offered, the queue being down for an hour delays the publish without losing it, the rows wait.
One saga walked through with its compensations in code
The order flow as an orchestrated saga reads as a state machine with a paired compensation table, and the code shape is worth printing because it makes the failure mode vocabulary concrete:
ORDER_SAGA = [
# (action, compensation)
(create_order, cancel_order),
(reserve_stock, release_stock),
(charge_payment, refund_payment),
]
def run_saga(command):
done = []
for action, compensate in ORDER_SAGA:
try:
action(command)
done.append((compensate, command))
except StepFailed as failure:
for compensate, cmd in reversed(done):
enqueue_with_retry(compensate, cmd)
return SagaResult.rolled_back(command, failure)
return SagaResult.completed(command)The compensation table is the design review's artifact, every step's undo named next to its do, and the step without a compensation, the email that was sent, the third party call that was made, is either moved before the compensated steps or accepted as unrecoverable with its name on the list. The compensation itself is a first class operation with its own retry policy and its own dead letter queue, and the monitoring watches the compensation queue with the seriousness of the payment queue, because a failed compensation is the reservation leak that quietly eats inventory.
The idempotent consumer that makes the delivery honest
The receiving end of every saga step completes the pattern, and the idempotency is usually one guarded insert:
def handle_order_created(event):
with db.transaction():
if exists(processed_events, event.id):
return # the duplicate is a no op
insert(processed_events, event.id)
reserve_stock(event.sku, event.qty)The processed events table grows and is pruned by age, the check and the insert are one transaction so the duplicate cannot slip between, and the consumer's contract is written on the wall, delivery is at least once, processing is exactly once, and the retry storms the broker produces on a bad day are absorbed as no operations rather than double reservations.
The failure modes the patterns accept on purpose
The patterns trade the lock's integrity for a set of named, designed failure modes, and the engineering is in making each one explicit. The intermediate state is the first, the saga's visible middle, and the design work is the business conversation, which states the user may see, the pending reservation, the payment in progress, and the interface vocabulary that makes them legible. The compensation failure is the second, the release that itself fails, and the answer is the retry with the dead letter path, the compensation that cannot complete becomes an operational queue with a human, because a system that silently drops compensations is a system that leaks reservations for months. The message order and the duplicate are the third, the at least once delivery of the outbox and the broker, and the idempotent consumer is the standard tool, the dedup window, the version check on the aggregate, the upsert that makes the second delivery free.
The semantic deadlock is the fourth and the one that deserves the design review, the cyclic dependency, service A's saga calls B and B's own saga calls A, and both wait. The design rules are the same ones the event driven discussion lands on, the dependency graph is a review artifact, cycles are resolved by merging the sagas or splitting the responsibility, and the retry policies carry jitter and bounds because the alternative is the thundering herd of compensations on a bad day. None of these failure modes is a surprise to the patterns' practitioners, and the difference between a saga architecture that runs and one that pages is whether the list above was written down before the first flow shipped.
The observability that makes the saga operable
The saga that cannot be observed cannot be operated, because its unit of failure is the flow, not the transaction, and the dashboards must answer the questions the incident channel will ask. The correlation identifier travels with every event and every step, stamped at the saga's start, propagated through the outbox, and logged by every consumer, so the question what happened to order 44172 is one query across the traces rather than an evening of log correlation. The saga state metric answers the health question, the count of flows by state, running, completed, compensating, dead lettered, and its shape over time is the early warning, a compensating count that rises on Tuesday afternoon is a downstream failure being absorbed correctly, and a dead letter count that rises is the one that pages.
The tracing integration is the pattern's natural fit with the OpenTelemetry work in this series, the saga's steps are spans in one trace, the compensations are spans too, and the trace of a rolled back order shows the whole story in one picture, three green steps, one red, three amber compensations. The outbox adds its own two metrics, the oldest unsent row's age and the sent lag, because the outbox that stops relaying turns every saga into a mystery of half steps, and its freshness is the pipeline's pulse. The observability list for a saga is short, correlation identifiers everywhere, state counts, outbox age, compensation queue depth, and the team that builds it with the first saga reuses it for every one that follows.
The decision rules for the consistency architecture
The architecture decision is per flow, not per system, and the rules compress into a short list:
- Keep local transactions local, the operation that fits one database is a transaction, not a saga;
- Route every cross service state change through an outbox, the publish is data and data is transactional;
- Choose choreography for flows inside one ownership group, orchestration for flows that cross department lines;
- Design every compensation as a real operation with a dead letter path, never as an assumption;
- Make every consumer idempotent and every event versioned, at least once delivery is the honest contract;
- Reserve XA for the rare homogeneous core that already runs it, and do not extend it to new boundaries.
The distributed transaction the modern system actually runs is the chain of local ones with its history in an outbox and its failures named and handled, and the teams that adopted the patterns early report the same arc, the first saga is a project, the third is a template, and the consistency that once needed a coordinator becomes the codebase's own vocabulary, compensations included.