Kafka event driven systems with duplicates ordering and lag in view

Kafka turns the event driven architecture from a style into an infrastructure, a durable, partitioned, high throughput log that services subscribe to at their own pace, and the systems built on it inherit a specific set of behaviors that the marketing material calls features and the on-call engineer meets as failure modes at three in the morning. The at least once delivery that brings the duplicate, the per partition ordering that the single topic assumption breaks, the consumer group rebalance that pauses processing exactly when the fleet is busy, the lag that grows quietly until the dashboard is a day behind. This article walks each of these as engineering rather than apology, the log's mechanics that produce the behaviors, the idempotent consumer that tames duplicates, the partition key discipline that preserves the order the business actually needs, and the lag management that keeps the dashboard honest.

The log mechanics that decide everything downstream

Kafka is a partitioned log, and every behavior this article covers falls out of two mechanical facts. The topic is split into partitions, ordered sequences of records on separate disks and brokers, and a producer's record lands in exactly one partition, chosen by the key's hash or round robin when no key is given. Ordering is therefore per partition, the records within one partition are stored and read in the order they were written, and there is no ordering across partitions, the record in partition two may be read before the record in partition one regardless of when each was written. The consumer group is the second fact, a set of consumers sharing the topic's partitions, each partition assigned to exactly one consumer in the group, and the position each consumer tracks is its own, stored on the broker, committed in intervals.

producer.send(ProducerRecord("orders",
    key=order.customer_id,      # same customer, same partition
    value=order_payload))

The key is the business decision hidden in a constructor argument. Same key, same partition, same order, and the order the business needs, this customer's events in sequence, is exactly the order the keyed partition provides, while the total event order the new team assumes does not exist at all. The partition count is the parallelism ceiling, a topic with six partitions processes at most six consumers per group, and the key that guarantees order also pins the load, the hot customer whose traffic saturates one partition while five sit idle, which is the arithmetic every topic design reviews.

The duplicate as the delivery model's first citizen

The at least once contract comes from the commit timing. The consumer reads a batch, processes it, and commits the position, and a crash between process and commit means the batch is delivered again to the next consumer, so every consumer in every Kafka system processes some records twice, by design, on the bad days when consumers restart. The idempotent consumer is the standard response and the outbox article already wrote its shape, the processed events table with the event identifier as its key, the check and the insert in the consumer's own transaction, the duplicate as a no operation. The Kafka native additions complete the toolkit, the idempotent producer that assigns sequence numbers per partition so the broker collapses the producer's own retries, and the transactional producer that commits offsets and writes output records atomically, the exactly once vocabulary within the Kafka boundary, worth adopting when the entire pipeline stays inside Kafka and the processing is a transformation between topics.

enable.idempotence=true      # producer retries collapse on the broker
isolation.level=read_committed  # consumer sees only committed transactions

The honest scope statement belongs in the design review, because exactly once inside Kafka is exactly once inside Kafka, the moment the pipeline touches a database, an API, a file, the boundary is crossed and the idempotent consumer is the contract again, and the team that shipped exactly once semantics into a database consumer built the duplicate that only appears in production.

The ordering that the key design actually delivers

The ordering question is per business entity, and the topic design that answers it is the key policy. The customer's events must be ordered, so the key is the customer identifier, and the partition count is set so the customer's traffic spreads but never splits. The order's events must be ordered, and the same question asks a harder decision when order and customer events share a topic, the key that orders one breaks the other, so the topics are split by entity or the key policy is written down as a contract with the consequences named. The cross topic ordering does not exist, the event in the orders topic and the event in the payments topic have no order relationship, and the consumer that depends on one is a consumer that will misbehave on the day the partitions align against it.

The rebalance is the ordering discussion's operational shadow. The consumer group rebalances when consumers join and leave, deploys, crashes, scaling events, and the rebalance revokes partitions and reassigns them, pausing processing for the group's cooperation window. The behaviors that follow are the classics, the deploy that pauses all consumers for the cooperation interval, the crash loop that turns processing into rebalancing, and the processing pause that the lag dashboard catches a minute late. The static membership, the cooperative rebalancing protocol and the deploy discipline, stop the old instance gracefully before the new one joins, shrink the window, and the consumer that assumes no rebalances is the consumer whose first deploy teaches the team otherwise.

The lag that grows until someone looks

Consumer lag is the distance between the log's end and the consumer's position, and it is the metric that tells the truth about the pipeline's health. The lag grows for three reasons, the consumer that is too slow, the consumer that is stopped, and the producer that spiked, and the response differs per reason, which is why the lag dashboard pairs the lag with the rate, the stable lag that tracks a stable rate is a consumer sized to its traffic, the growing lag with a stable rate is a consumer that needs another partition or another instance, and the lag spike with a producer spike is the fleet absorbing a batch, which the alert should acknowledge rather than page.

kafka-consumer-groups.sh --bootstrap-server kafka:9092 \
  --describe --group order-processor

The lag alerting that works pages on the trend rather than the absolute, the lag that grows for ten minutes is the incident, the lag that exists is Tuesday, and the threshold is calibrated per consumer per traffic pattern. The dead letter and the retry complete the lag management, the record that fails processing goes to a retry topic and then a dead letter queue rather than blocking the partition, because the consumer that parks on a poison record is the pipeline that stops for one bad message, and the dead letter queue is the operational queue with a human and a review process, not the trash folder the name suggests.

One topic designed around its order requirement

The key decision reads best as one worked example. The account transactions topic must process each account's events strictly in order, balances depend on the sequence, while the overall throughput wants parallelism across thousands of accounts. The design is the account identifier as the key, partitions at twelve, sized for the consumer fleet's parallelism ceiling with headroom, and the order guarantee arrives per account because the hash pins every account to one partition:

topic: account-transactions
key:   account_id           # per account order, guaranteed
partitions: 12             # parallelism ceiling, headroom for growth

The consequences are named in the review, the hot account is single threaded by design and its throughput is the partition's throughput, the fleet scales to twelve consumers and no further without repartitioning, and repartitioning rehashes every key, which breaks order for the accounts that move during the migration window. The cross entity order that the business assumed, the account event and the customer event in sequence, is designed out loud, the customer topic exists separately and the consumer that needs both subscribes to both and reconciles by timestamp, a design decision the review records rather than the runtime discovering. The topic that was designed around its order requirement runs for years, and the topic that was designed around its throughput discovers its order requirement in the balance defect that finance files.

Throughput backpressure and the consumer that cannot keep up

The lag discussion's practical half is the consumer that is consistently slower than its producer, and the response ladder is worth writing down. The first rung is parallelism, more consumers up to the partition count, the cheapest scaling the log offers. The second is batching, the consumer that processes one record at a time becomes the consumer that processes five hundred per loop with one offset commit, the throughput multiplier that the database batch inserts already taught this series. The third is the processing split, the expensive transformation moves to its own consumer group with its own topic, the pipeline stage that scales independently, and the fourth rung is the capacity conversation, more partitions, which is the migration with the rehashing consequence from the example above, priced and scheduled rather than performed on call.

The backpressure that the log gives for free is the property the whole design rests on, the slow consumer does not slow the producer, the log absorbs and the lag grows, and the growth is visible rather than viral, which is the log's advantage over the request driven coupling it replaced. The capacity decision is therefore always informed, the lag dashboard is the queue depth that says exactly how far behind and how fast, and the consumer that is sized against its lag trend is the consumer that never surprises anyone, which is the entire operational promise of the durable log architecture.

The schema contract and the evolution discipline

The pipeline's data has a schema, and the unmanaged schema is the pipeline's slow disaster, the producer that renames a field and the consumers that deserialize it into silent nulls, the breaking change that nobody detected because the contract lived in a wiki. The schema registry is the discipline's tooling, the producer registers the schema, the registry enforces the compatibility policy per topic, and the consumer pulls the exact contract to generate its code:

# compatibility modes
BACKWARD:  new schema reads old records
FORWARD:   old schema reads new records
FULL:      both, the mode for topics with many consumers

The evolution discipline is the versioning conversation the architecture discussions in this series keep having, additive changes by default, the deprecation window for the removed field, the major topic for the breaking change, and the registry is the enforcement that turns the policy from a review memory into a build failure. The teams that skip the registry meet the schema drift on the morning the producer team ships confidently, and the incident is the consumer that crashed at scale.

The working rules for the pipeline

The mechanics, the duplicates and the lag assemble into an operating discipline, and the list is short:

1. Choose the partition key per business entity, write the key policy down, and size partitions for the parallelism plus the growth;
2. Build every consumer idempotent from its first message, the processed events table is the standard shape;
3. Alert on lag trends, not absolute values, and pair the alert with the rate;
4. Route poison records to a retry and a dead letter topic, never park the partition;
5. Register schemas and enforce compatibility, the contract is a build failure or it is a wiki;
6. Treat rebalances as routine, static membership, graceful deploys, and processing that assumes the pause.

The log is an honest infrastructure, it does what its mechanics say, and the systems that run well on it are the ones that read the mechanics as contracts, duplicates arrive, order exists only where the key puts it, and the lag tells the truth. The teams that design to those contracts build pipelines that survive their own scale, and the teams that design to the brochure build pipelines that discover the mechanics in production, one duplicate, one reorder and one rebalance at a time.