A customer got charged twice for one order. Nothing ran twice on purpose: the payment worker charged the card, then crashed before it could ack the message — so the broker, doing exactly what it promised, delivered the message again.

The duplicate wasn’t the bug. The handler that wasn’t built for a duplicate was.

At-least-once is a promise, not a glitch

Almost every queue — and every retried HTTP call — is at-least-once. A handler MAY see the same message more than once: after a crash, a release, a redelivery, or a client that timed out and retried. Exactly-once across a network is impractical without coordination the broker can’t give you cheaply, so nobody serious offers it.

That reframes the problem. The redelivery isn’t a failure to eliminate; it’s a guarantee to design around. (It’s the same trade the queue makes when it retries a failed job instead of losing it.)

Idempotent means “twice equals once”

An operation is idempotent if applying it many times has the same effect as applying it once — f(f(x)) = f(x).

  • “Set order status to paid” is idempotent.
  • “Increment balance by 10” is not — twice is +20.
  • “Send the welcome email” is not — twice is two emails.

The whole job is turning the second kind of operation into the first.

The idempotency key comes from the sender

You need a stable identity for this operation, and it must be decided by the sender, not the receiver:

  • For a queue message, it’s the per-message id the producer mints — one id per message, distinct from the trace/correlation id (which spans many messages).
  • For an HTTP write, it’s a client-generated Idempotency-Key header — the Stripe model.

Two rules keep the key honest:

  • Don’t derive it from the payload alone. Two legitimately identical requests — the same customer buying the same item twice — would collide and the second would be silently dropped.
  • Don’t let the receiver mint it. The receiver can’t tell a retry from a brand-new call; only the sender knows “this is the same operation I tried before.”

Dedupe: remember what you’ve already done

The handler does one check before the work: has this key been processed? If yes, skip and ack. If no, do the work, then record the key. That’s the entire pattern — the weight is in where the record lives.

The crash between the work and the record

The hard part is the gap between doing the work and recording the key. Two shapes, and the right one depends on the side effect:

  • Seen-set — record after success. Store the key in Redis or a table once the handler returns. Cheap and broadly applicable. The window: crash after the side effect but before recording, and a redelivery reprocesses. That’s fine when the side effect is itself idempotent — an UPSERT, a “set to paid”.
  • Transactional — record with the work. Write the idempotency record in the same database transaction as the business change. No window: the key and the effect commit or roll back together. The cost: the effect must be a DB write you control, and the key store must be that same database — not a separate Redis. This is what kills the double-charge for real.

So the side effect chooses for you: a row you own → transactional; a third-party call you can’t enlist in your transaction (email, a charge) → seen-set, and make the downstream idempotent too by forwarding your key as its Idempotency-Key.

Concurrent duplicates need a constraint, not a check

Two deliveries of the same key racing before either records it slip through a “check then write” — that sequence isn’t atomic. A unique constraint on the key column is: the second insert becomes a conflict you catch and treat as “already handled.” This is the same shape as the gaps a race opens in sequential numbering — the database, not the application, is where uniqueness is actually enforced.

When do you not need any of this?

If the handler is already idempotent — a pure UPSERT keyed by a stable id, a “set field to value” — you may need nothing at all. Reach for a key plus dedupe when the operation has a non-idempotent side effect (money, email, an external POST) or accumulates (increment, append). Spend the complexity only where a second run does real damage.

At-least-once isn’t the part you fix; it’s the part you accept. Build the handler so the second delivery is a no-op, and redelivery stops being an incident and goes back to being what it is — the broker keeping its promise.

See also: the BabelQueue idempotency spec pins down where the key comes from and how dedupe behaves, and the idempotency-payments example is exactly this double-charge case made runnable.