Idempotent Event Processing With Azure Event Hubs
Part 1
Why Azure Event Hubs consumers receive duplicate events (producer retries, batched checkpoints, crashes, partition rebalancing) and what the literature recommends: at-least-once delivery combined with idempotent processing for effectively-once outcomes.

When you consume events from Azure Event Hubs, the same event can reach your handlers more than once.
This means most consumer processing logic should be idempotent: handling an event a second time should leave the system in the same state as handling it once.
Idempotency here is a property of the processing logic you write, and not necessarily a feature of Azure Event Hubs.
This first post covers why idempotency matters and the theory behind it. The upcoming second part will share concrete techniques that have been useful to me for implementing idempotent event processing, stay tuned 😉
Why Idempotency?
Event Hubs consumers typically get an at-least-once event delivery guarantee: an event can be delivered and processed one or more times; you shouldn’t expect exactly once.
Duplicates can appear at two points of the pipeline: when sending and when receiving.
Duplicates While Sending
A producer that doesn’t receive an acknowledgment can’t tell whether the service stored the event. Typically, the SDK retry policy sends it again. If the first attempt did succeed and only the acknowledgment was lost, the event hub now holds two copies.
At runtime, the two copies are separate events with their own offsets and sequence numbers. The consumer has no built-in way to know they represent the same business fact and should rely on application specific keys to determine uniqueness.
Duplicates While Receiving
As discussed in the Azure Event Hubs Checkpoints & Rewinding post, consumers track progress with a checkpoint stored in a checkpoint store. A new owner of a partition resumes from the last checkpoint (see also Balance partition load across multiple instances).
We also discussed why checkpointing at every event is not a good idea: each update has a cost, so checkpoints are typically batched.
The consequence is that some events can be processed but not yet checkpointed, and those get processed again.

Common situations include:
- Batched or interval checkpoints: checkpointing every N events or every few seconds is cheaper, but widens the window of events that get replayed after a failure.
- No checkpoint on exit: if the processor stops without writing a final checkpoint, the events handled since the last one are replayed on restart.
- Crash: the process dies between handling an event and checkpointing it.
- Partition ownership changes: when instances join or leave (during auto-scale for example), partitions are rebalanced. The new owner starts from the last checkpoint, while the previous owner may still have been working on events past it.
Does Checkpointing At Every Event Solve This?
Checkpointing after each event narrows the replay window at a cost but doesn’t close it. A crash can still occur in the event handler before the checkpoint operation.
At runtime, the sequence looks like this:
- The handler performs an operation.
- The process crashes before
UpdateCheckpointAsynccompletes. - The next owner resumes from the previous checkpoint and handles the same event again.
One might think about checkpointing before handling the event, this drastically reduces duplicates but introduces a bigger risk of missing events.
For most workloads, I would prefer duplicates over losses, and then make the handler tolerate them.
Idempotent Processing Theory and State of The Art
Idempotency is a widely discussed (dare I say classic?) subject in computer science and software engineering. Before looking at concrete techniques, let’s look at some reference sources that discuss the matter.
The term comes from mathematics, where an idempotent function produces the same result when applied to its own output: f(x) = f(f(x)). In messaging, this translates into a message that has the same effect whether the receiver handles it once or several times.
The following are the resources I found most useful, from the general pattern to the Azure specific guidance.
Enterprise Integration Patterns
Enterprise Integration Patterns (EIP) is a classic messaging book. It describes the Idempotent Receiver pattern, which starts from an observation: even when a sender sends a message once, the receiver may receive it more than once.
The proposed solution is to design a receiver that can safely receive the same message multiple times. EIP identifies two primary means:
- Explicit deduplication: remove duplicate messages, typically by remembering which ones were already handled.
- Message semantics: define messages so that applying them again is harmless, for example “set the quantity to 5” instead of “add 1 to the quantity”.
Each technique in the upcoming part 2 will indicate which of these two categories it belongs to. The pattern is also related to Guaranteed Delivery: a messaging system that persists messages to avoid losing them will, in some failure cases, deliver them again.
Azure Architecture Center and Azure Functions
The Idempotent Consumer pattern in the Azure Architecture Center is the same idea applied to brokers such as Event Hubs, Service Bus, Kafka and RabbitMQ. It states that exactly-once delivery is impractical across a distributed system, and that at-least-once delivery combined with a consumer that ignores duplicates gives effectively-once processing.
It also turns the deduplication option into an operational checklist:
- Choose a stable deduplication key that survives redelivery, such as a producer-assigned message identifier or a business-level idempotency key.
- Decide where to store processed keys: a dedicated table (sometimes called an inbox) or the business entity itself.
- Commit the key and the side effects in the same transaction, otherwise a crash between the two leaves a failure window.
- Guard against concurrent duplicates with a uniqueness constraint, since a check-then-process flow can race when competing consumers receive the same message.
Two other Microsoft Learn articles complement it:
- Designing Azure Functions for identical input lists defensive behaviors such as verifying that data exists before a delete, checking that it doesn’t exist before a create, concurrency controls and data freshness validation. Most of these are semantic techniques in EIP terms.
- Resilient Azure Event Hubs and Azure Functions design discusses idempotency specifically in the Event Hubs context, which is the same guidance linked earlier in this post.
Designing Data-Intensive Applications
Designing Data-Intensive Applications by Martin Kleppmann covers the design of data systems in general. Its chapter on stream processing asks a question that applies directly to Event Hubs consumers: if processing can be repeated after a failure, how do you get the effect of processing each event exactly once?
The answer I found most practical is idempotence: repeating an operation has the same effect as running it once, so a retry is harmless. Extra metadata can turn an operation that isn’t naturally idempotent into one that is.
The chapter also discusses atomic commit protocol, where the output of processing and the acknowledgment of the input (such as the offset) succeed or fail together in one transaction. In my opinion, it is more about implementing transactions across systems than about making processing repeatable.
In practice, this metadata is a marker stored with the state, written in the same atomic operation as the state change. Two kinds of markers are useful with Event Hubs:
- A position: the sequence number assigned by Event Hubs when the event was enqueued in a partition. It tells you whether this event is newer than the last one applied.
- An application specific key: an identifier set by the producer, such as
EventData.MessageId, that uniquely identifies the logical event. It tells you whether this event was already applied.
The position works well when events are processed in order: a replayed event carries a sequence number lower than or equal to the stored one and is ignored. It only makes sense within an ordered stream, and Event Hubs orders events per partition, not across the event hub.
The application specific key covers what the position can’t. As discussed in Duplicates While Sending, a producer retry creates a second event with its own sequence number, so the position looks new. The consumer instead records the keys it already processed, ideally in the same transaction as the side effect.
Kleppmann also notes that idempotence may require fencing when a partition moves from one node to another, so that the previous owner can’t keep applying updates. In Event Hubs terms, this maps to the partition ownership changes discussed earlier.
The conclusion matches the Azure Architecture Center guidance: idempotent processing on top of at-least-once delivery achieves exactly-once effects with a small overhead. The delivery stays at-least-once; only the observable outcome becomes exactly-once.
Additional Resources
For a more theoretical angle, Pat Helland’s Idempotence Is Not a Medical Condition (ACM Queue) discusses why at-least-once messaging pushes the responsibility of handling duplicates to the application.
Closing Thoughts
This post covered why Event Hubs consumers must tolerate duplicates, and what the literature recommends: at-least-once delivery combined with idempotent processing gives effectively-once outcomes.
In the upcoming part 2, I will walk through concrete techniques, from the most generic to the most domain specific, using a runnable sample. Each one comes with the failure scenarios it handles and the ones it doesn’t.