Idempotency and job queues: at-least-once delivery, retries and exactly-once effects

Key points

• At-most-once loses messages and at-least-once duplicates them; with retries and timeouts, at-least-once is what you get by default.
• Exactly-once effects come from idempotent processing: a dedupe record with a unique constraint, written in the same transaction as the side effect.
• Idempotency keys let a client retry a POST without doing the work twice: the server stores the first response and checks that a retry carries the same request.
• The transactional outbox writes an event to a table in the same transaction as the data, and a relay publishes it, sometimes more than once.
• Retry transient failures with backoff, cap the attempts, and move poison messages to a dead-letter queue that someone watches.
• A handler slower than its visibility timeout can run at the same time as its own redelivery, so idempotency matters even when nothing crashes.

Delivery semantics: at most once, at least once, exactly once

Every queue sits on a network that can fail between “the work is done” and “the queue knows it is done”. Where the acknowledgment happens decides what you get. Kafka’s documentation defines the three guarantees plainly: at most once, where messages may be lost but are never redelivered; at least once, where messages are never lost but may be redelivered; and exactly once, where each message is processed once and only once.

At-least-once is the usual default. Kafka’s documentation says it guarantees at-least-once delivery unless configured otherwise, and the Amazon SQS documentation says a standard queue may deliver a message more than once. Workers built on a database or Redis behave the same way: a worker that dies holding a job gives it back, and the next worker runs it again. The wider picture of queues and workers is in background jobs and queues, and webhooks, which providers retry, raise the same problem (see API vs webhook).

Why “exactly once” is really at-least-once plus idempotency

A broker can deduplicate what it controls. Kafka’s idempotent producer, added in version 0.11, has the broker assign each producer an id and drop resent messages by sequence number, and Kafka transactions write to several partitions so that either all of the messages are written or none are. SQS FIFO queues drop a resend that carries the same deduplication id within a five-minute interval. These help, but they stop at the broker’s edge.

The side effect you care about, a row in your database, an email, a charge, happens outside the broker. Kafka’s documentation says exactly-once delivery to an external system generally needs that system’s cooperation, and suggests storing the consumer’s position in the same place as its output so both change together. That is the general rule: exactly-once effects come from an idempotent handler, so that a second delivery finds the work done and does nothing.

Some operations are idempotent by nature: setting a status to shipped, upserting a row by its natural key, deleting a record. Others are not: adding to a balance, appending a line, sending a message. Rewrite the first kind where you can (“set the plan to Pro” rather than “upgrade one tier”) and guard the second kind with a dedupe record.

An idempotent handler with a dedupe table

Give each message a stable id, assigned by the producer when the job is created, not by the queue at delivery. In the handler, insert that id into a table with a unique constraint, in the same database transaction as the work. If the insert conflicts, an earlier delivery already committed the work, so skip it and acknowledge. The table can be as small as processed_messages (message_id text primary key, handled_at timestamptz).

# PostgreSQL SQL. `db` is a connection whose `with` block is one
# transaction (psycopg works this way).
DEDUPE = """
    INSERT INTO processed_messages (message_id, handled_at)
    VALUES (%s, now())
    ON CONFLICT (message_id) DO NOTHING
"""
CREDIT = "UPDATE accounts SET balance = balance + %s WHERE id = %s"

def handle(db, message) -> None:
    with db:  # commit both statements or neither
        cur = db.cursor()
        cur.execute(DEDUPE, (message["id"],))
        if cur.rowcount == 0:
            return  # seen before: the credit is already applied
        cur.execute(CREDIT, (message["amount"], message["account_id"]))
    # Acknowledge only after the commit. A crash before the acknowledgment
    # means a redelivery, and the dedupe row turns it into a no-op.

• The unique constraint does the work, not a SELECT before the insert. Two workers that check first and insert second can both see “not processed”; with the constraint, only one insert wins. PostgreSQL’s ON CONFLICT DO NOTHING skips the conflicting row, and the row count of zero tells the handler it lost.
• This works only when the side effect lives in the same database as the dedupe table. For an email or a call to another API, pass an idempotency key downstream, or write an outbox row and let a single sender handle it; both are below.
• Prune the table by age, but keep each id longer than the longest possible redelivery, including manual replays from a dead-letter queue.

Idempotency keys for APIs

The same problem appears at an API. A client sends a POST, the connection drops, and the client cannot tell whether the order was created. An idempotency key lets it retry safely: the client generates a unique key for each logical operation and sends it with every attempt, and the server records the key with the result and replays that result for a retry.

Stripe’s documentation is a good reference design. It suggests V4 UUIDs or another random string with enough entropy, accepts keys of up to 255 characters, and saves the status code and body of the first request once execution begins, 500 errors included, returning them for any retry with the same key. It compares a retry’s parameters with the original and returns an error if they differ. Keys can be pruned once they are at least 24 hours old. Requests that fail validation, or that conflict with a request still running, are not saved, so they can be retried. Every POST accepts a key; GET and DELETE are idempotent by definition, so a key has no effect there.

The IETF draft “The Idempotency-Key HTTP Header Field” would standardize this. As of November 2026 it is at version 08, published in October 2026, an active Internet-Draft of the HTTPAPI working group and not yet an RFC. It recommends UUIDs as keys and suggests three errors: 400 when a required key is missing, 409 when a retry arrives while the original request is still being processed, and 422 when a key is reused with a different payload. It also describes an optional fingerprint of the request, stored with the key, to detect that reuse.

1. Read the key. If the endpoint requires one and it is missing, return 400.
2. Insert the key, scoped to the caller (the user or API key), with a fingerprint of the request and a status of “in progress”. A unique constraint on the caller and key makes this insert the lock.
3. On a conflict, load the stored row. A different fingerprint means 422, “in progress” means 409, and a stored response is returned exactly as it was.
4. Do the work, then store the status code and body against the key, in the same transaction as the work where you can.
5. Expire keys after a period you document, so clients know how long a retry stays safe.

Scope keys to the caller, or one client can replay another’s response. Keep personal data out of the key itself; Stripe’s documentation warns against using email addresses or personal identifiers. Clients should retry with backoff and jitter, the same schedule used against a rate limiter (see designing a rate limiter).

The transactional outbox

A handler that writes to the database and then publishes an event has a dual-write problem. Crash between the two, and the database changed but nobody heard about it. Publish first, and a rolled-back transaction leaves an event about something that never happened. Two-phase commit across the database and the broker is often unavailable or unwanted.

The transactional outbox pattern, as Chris Richardson describes it on microservices.io, inserts the outgoing message into an outbox table in the same transaction as the business data. A separate message relay reads the outbox and publishes to the broker, either by polling the table or by tailing the database’s transaction log.

import json
import uuid

def create_order(db, order_id, customer_id, total_cents) -> None:
    event = json.dumps({"order_id": order_id, "total_cents": total_cents})
    with db:  # the order and its event commit together, or not at all
        cur = db.cursor()
        cur.execute("INSERT INTO orders (id, customer_id, total_cents) VALUES (%s, %s, %s)",
                    (order_id, customer_id, total_cents))
        cur.execute("INSERT INTO outbox (id, topic, payload) VALUES (%s, %s, %s)",
                    (str(uuid.uuid4()), "order.created", event))

def relay_once(db, publish, batch=100) -> None:
    with db:
        cur = db.cursor()
        cur.execute("""SELECT id, topic, payload FROM outbox WHERE published = false
                       ORDER BY created_at LIMIT %s FOR UPDATE SKIP LOCKED""", (batch,))
        for row_id, topic, payload in cur.fetchall():
            publish(topic, payload, message_id=row_id)  # a crash after this means a resend
            cur.execute("UPDATE outbox SET published = true WHERE id = %s", (row_id,))

• The relay can publish a message more than once, for example when it crashes after publishing but before marking the row; microservices.io says so directly. That is why the outbox row’s id travels with the message as message_id: consumers dedupe on it with the table from earlier.
• FOR UPDATE SKIP LOCKED lets several relays run without picking the same rows. PostgreSQL’s documentation notes that skipping locked rows gives an inconsistent view of the data, unsuitable for general work but useful for consumers of a queue-like table.
• If consumers need events in order, publish in created_at order per entity, with a single relay or with relays partitioned by entity id.
• Delete or archive published rows, or the outbox grows forever.
• The consumer-side mirror image is sometimes called an inbox: the incoming message id recorded in the same transaction as the work. It is the dedupe table from the previous section.

Retries, poison messages and dead-letter queues

Failures come in two kinds. Transient ones, such as a timeout, a lock conflict or a 429 from a downstream API, usually succeed on a later try. Permanent ones, such as a malformed payload or a reference to a deleted record, fail every time. A message that fails every time is a poison message: retried forever, it burns capacity, and in an ordered queue it blocks everything behind it.

• Retry transient failures with exponential backoff and jitter, so a struggling downstream service gets room to recover instead of a synchronized wave of retries.
• Classify errors in the handler. A validation error should go straight to the dead-letter queue, not through every retry first.
• Cap the attempts. In SQS, a redrive policy’s maxReceiveCount sets how many times a message can be received before it moves to the dead-letter queue. AWS notes that a value as low as 1 moves a message after a single failure, so set it high enough for real retries.
• Treat the dead-letter queue as an inbox for people: alert when it is not empty, record why each message failed, then fix the cause and redrive. SQS can move messages from a dead-letter queue back to the source queue.
• Keep dead-lettered messages longer than the source queue does. In an SQS standard queue a message’s expiry still counts from its original enqueue time, so AWS recommends a longer retention period on the dead-letter queue.
• A redrive is a redelivery. The message passes through the same idempotent handler, so the dedupe table must still remember its id.

Visibility timeouts and leases

Queues that hand each message to one consumer at a time do it with a lease. In SQS this is the visibility timeout: a received message stays in the queue but is hidden from other consumers, and if it is not deleted before the timeout ends, it becomes visible again. The default is 30 seconds, ChangeMessageVisibility extends or shortens it, and the maximum is 12 hours from when the message was first received.

The trap is a handler slower than its lease. The message reappears, a second worker picks it up, and two copies run at the same time with no crash involved. AWS also says that, because of the at-least-once model, a standard queue does not guarantee against a second delivery within the visibility timeout. So:

• Set the timeout above the handler’s worst normal run time, and for long jobs extend it with a heartbeat while work continues, as the SQS documentation recommends.
• Split long jobs into steps that each finish well inside the lease, each with its own id.
• Keep the handler idempotent anyway: the dedupe table’s unique constraint turns the slower copy into a no-op.
• For side effects outside your database, such as a charge or an outgoing email, pass the message id to the downstream API as its idempotency key, so the downstream service does the dedupe.

Ordering

Ordering and parallelism pull against each other. A standard queue with many consumers gives no order at all. Ordered systems give order within a partition or a group: Kafka within a partition, and SQS FIFO within a message group, where the next message in a group is not available until the one in flight is deleted or its visibility timeout runs out.

• Ask for the order you need, which is usually per entity (per account, per order), not global. Use the entity id as the partition key or group id, and different entities still process in parallel.
• Retries break order. A message retried later can land after newer ones, and AWS advises against a dead-letter queue on a FIFO queue when the exact order of operations must not break.
• Make handlers tolerant of order where you can: carry a version number on each event and ignore any event older than the version already applied. That check is often simpler than strict ordering.
• With at-least-once delivery, a duplicate can arrive after later messages. The dedupe record and the version check together handle it.

A checklist for a reliable worker

1. Assume every message arrives at least twice, and test the handler by delivering each one twice.
2. Give every job and event a stable id at creation, and dedupe on it under a unique constraint in the same transaction as the work.
3. Publish events through an outbox, not straight after a commit.
4. Accept idempotency keys on every POST that creates or changes something, scoped to the caller and stored with the response.
5. Retry transient errors with backoff and jitter, send permanent ones to a dead-letter queue, and alert on it.
6. Size leases above the slowest normal run, extend them for long jobs, and keep order per entity only where it matters.

Related deep dives: caching strategies, where invalidation events often travel through the same queues, and database indexing explained, since a dedupe table is only as fast as its unique index.

BYD Tech builds web apps and APIs, background jobs included: the code lives in the client’s GitHub from day one, and AI-assisted work is reviewed by an engineer.