System design: a reliable webhook delivery system

Key points

• Promise at-least-once delivery and say so: receivers must dedupe on the event id, and most providers do not guarantee order.
• Write the event in the same transaction as the data change (an outbox), then fan out to one delivery record per endpoint.
• Retry over days, not seconds: capped exponential backoff with jitter, a deadline, then a dead letter and a manual replay.
• Sign the id, a timestamp and the raw body with HMAC-SHA256; receivers compare in constant time and reject old timestamps.
• Customer URLs are untrusted input: resolve them, refuse private and metadata addresses, connect to the checked IP and never follow redirects.
• Keep a delivery log customers can read: every attempt, its status code, latency and the next retry time.

Requirements: what a webhook system promises

A webhook is an HTTP POST your system sends to a URL your customer chose, when something happens on your side. The customer’s server may be slow, down for an afternoon, behind a broken certificate, or not theirs at all. The design work is deciding what you promise despite that, and building the parts that keep the promise. The basic trade-off between polling an API and receiving webhooks is in API vs webhook; this post is about the sending side.

Exactly-once delivery over HTTP is not on the list. If a worker sends a request and the connection drops before the response arrives, the sender cannot know whether the receiver processed it, so it sends again. Stripe’s documentation says plainly that endpoints might occasionally receive the same event more than once. The same reasoning, for queues in general, is in idempotency and job queues.

Ordering deserves an explicit decision. Strict order per endpoint means one failing event blocks every event behind it until it succeeds or is dropped, which is head-of-line blocking across hours of retries. Most providers choose not to guarantee it: Stripe’s documentation says it does not guarantee delivery in the order events were generated, and tells receivers not to depend on it. A cheaper option is to put a version or sequence number on each event and let receivers ignore anything older than what they have already applied.

Architecture: outbox, queue, delivery workers

1. The business change and its event are written in one database transaction: the order row and an outbox row with the event type, a unique event id and the body. A crash cannot leave one without the other.
2. A relay reads new outbox rows and, for every endpoint subscribed to that event type, creates a delivery record (event id, endpoint id, attempt count, status, next attempt time) and enqueues it.
3. Delivery workers take a delivery, load the endpoint’s URL and secret, sign the request and POST it with a strict timeout.
4. A 2xx response marks the delivery succeeded. Anything else, including a timeout, a TLS error or a redirect, schedules the next attempt or, past the deadline, marks it failed.
5. Every attempt is written to a delivery log the customer can read in a dashboard or through an API.

The outbox is the part teams skip and regret. Publishing to the queue straight after the commit loses the event if the process dies between the two; publishing before the commit sends events about changes that rolled back. The pattern, and why its relay can itself publish twice, is covered in idempotency and job queues.

• Timeouts: the Standard Webhooks specification recommends a request timeout of 15 to 30 seconds. GitHub expects a 2xx within 10 seconds and counts a slower response as a failure. Short timeouts protect your worker pool; receivers should acknowledge first and process later.
• Status codes: treat 2xx as success and everything else as failure. Standard Webhooks and Stripe both treat a 3xx as a failure rather than following it; the customer should update the URL instead.
• Special cases from Standard Webhooks: a 410 Gone means the receiver no longer wants webhooks, so disable the endpoint; 429, 502 and 504 mean slow down; a retry-after header, when present, should inform the next attempt.
• Body size: Standard Webhooks suggests keeping bodies small, ideally under about 20 KB, and linking to large assets instead of embedding them.

Retries: exponential backoff, jitter and a deadline

Most delivery failures are temporary: a deploy, an overloaded server, an expired certificate someone fixes in the morning. Retrying immediately and often wastes effort and hits a struggling receiver when it is weakest. Retrying with growing gaps covers long outages with few attempts. Stripe’s documentation says it retries for up to three days with exponential backoff in live mode. The Standard Webhooks example schedule retries after 5 seconds, 5 minutes, 30 minutes, then 2, 5, 10, 14, 20 and 24 hours, a little over three days in all. GitHub is the counterexample: its documentation says it does not automatically redeliver failed deliveries, and suggests a scheduled script that finds failures through its REST API and redelivers them.

import random

def retry_delays(base=30.0, cap=6 * 3600.0, horizon=72 * 3600.0, max_attempts=30):
    """Seconds to wait before each retry: exponential, capped, jittered."""
    delays, elapsed = [], 0.0
    for n in range(max_attempts):
        ceiling = min(cap, base * 2 ** n)  # 30s, 1m, 2m, 4m ... capped at 6h
        delay = ceiling / 2 + random.uniform(0, ceiling / 2)  # "equal jitter"
        if elapsed + delay > horizon:
            break  # out of time: mark the delivery failed and dead-letter it
        delays.append(delay)
        elapsed += delay
    return delays

# Ten calls gave 23 to 25 retries each, spanning 67.5 to 71.6 hours.

• Jitter matters for webhooks because failures are correlated. When a large receiver goes down, thousands of deliveries fail in the same second; without jitter they all retry in the same second too. Marc Brooker’s AWS Architecture Blog post compares jitter strategies; the code uses what he calls “equal jitter”, which keeps a floor of half the ceiling so a retry never comes back almost at once.
• Store the next attempt time on the delivery record and let a scheduler pick up due deliveries. Do not sleep inside a worker for six hours.
• Generate a fresh timestamp and signature for every attempt. Stripe does this, which keeps retries inside the receiver’s replay window.
• If a receiver sends retry-after, use the later of that time and your own schedule.

Failing endpoints, dead letters and manual replay

Two kinds of failure need different handling. A delivery that runs out of retries is one event that did not arrive. An endpoint that fails every delivery for days is a broken integration: retrying each of its events through the full schedule burns worker time and delivery log space for nothing.

• Dead-letter, do not delete. A delivery past its deadline moves to a failed state and stays visible in the customer’s log with the last error. The event itself stays in your store, so it can be sent again.
• Track health per endpoint: the recent success rate and the time since the last success. Standard Webhooks suggests notifying the customer through another channel, such as email, when an endpoint fails consistently over a long period, and considering disabling it. A 410 Gone is an explicit request to disable it.
• Disable with notice. Tell the customer before and when an endpoint is disabled, keep the events, and let them re-enable it and replay what they missed.
• Offer manual replay of one event and of a time range. Stripe’s Dashboard can resend an event for up to 15 days after it was created, and its CLI for up to 30 days. A replay is a new attempt of the same event, with the same event id, so a receiver that dedupes treats it correctly.
• Replays go through the same rate limits and fairness rules as live traffic, so a customer replaying a week of events does not crowd out everyone else.

Signing: HMAC-SHA256 over the id, a timestamp and the body

A webhook URL is often guessable and sometimes leaked, so the receiver needs proof that a request came from you. The usual answer is HMAC, defined in RFC 2104: a keyed hash computed with a secret shared between sender and receiver. The sender computes HMAC-SHA256 over the request and puts the result in a header; the receiver recomputes it with its copy of the secret and compares. A matching signature shows the sender held the secret and the signed bytes were not changed.

What gets signed decides what is protected. GitHub signs the body only: the X-Hub-Signature-256 header carries sha256= and a hex HMAC of the body, with no timestamp, so a captured request stays valid forever. Its documentation suggests using the X-GitHub-Delivery id to detect a replay. Stripe signs the timestamp, a full stop and the body, sends both as t= and v1= in the Stripe-Signature header, and its libraries reject timestamps more than 5 minutes from the current time by default. The Standard Webhooks specification signs msg_id.timestamp.body and sends three headers: webhook-id, webhook-timestamp and webhook-signature, where the signature is v1, followed by a base64 HMAC-SHA256.

import base64
import hashlib
import hmac
import time

TOLERANCE = 5 * 60  # seconds either side of the receiver's clock

def sign(secret: bytes, msg_id: str, timestamp: int, body: bytes) -> str:
    to_sign = f"{msg_id}.{timestamp}.".encode() + body  # the raw body bytes, as sent
    mac = hmac.new(secret, to_sign, hashlib.sha256).digest()
    return "v1," + base64.b64encode(mac).decode()

def verify(secret: bytes, headers: dict, body: bytes) -> bool:
    msg_id = headers["webhook-id"]
    timestamp = int(headers["webhook-timestamp"])
    if abs(time.time() - timestamp) > TOLERANCE:
        return False  # too old, or from the future: treat it as a replay
    expected = sign(secret, msg_id, timestamp, body)
    # During a secret rotation the header carries several signatures, space-separated.
    return any(hmac.compare_digest(expected, s)
               for s in headers["webhook-signature"].split())

• Sign and verify the raw bytes. Stripe’s documentation warns that a framework which parses and re-serializes the JSON body breaks verification; one changed space changes the HMAC.
• Compare in constant time, with hmac.compare_digest in Python or crypto.timingSafeEqual in Node.js. GitHub’s documentation says never to use a plain ==.
• The timestamp tolerance is the replay window. Five minutes, Stripe’s library default, allows for clock skew without letting a captured request work for long. Stripe warns that a tolerance of 0 disables the check entirely in its libraries. Pair the window with event-id dedupe on the receiver to stop replays inside it.
• Use one secret per endpoint, so a leak affects one integration. Standard Webhooks generates secrets of 24 to 64 random bytes, base64-encoded with a whsec_ prefix; RFC 2104 discourages keys shorter than the hash output.
• Support rotation: during a rotation, send one signature per active secret in the same header. Stripe keeps an old secret valid for up to 24 hours after a roll, and Standard Webhooks puts several signatures in one space-separated header for the same reason.
• Version the scheme. Receivers should ignore schemes they do not know; Stripe tells them to ignore everything but v1 to prevent downgrade attacks. Standard Webhooks also defines v1a, an asymmetric ed25519 signature, so receivers verify with a public key and hold no shared secret.

SSRF: calling URLs your customers choose

A webhook sender is a server-side request forgery machine by design: it makes HTTP requests from inside your network to addresses an outsider typed in. Without checks, a customer can register http://169.254.169.254/ and have your workers fetch cloud credentials from the metadata service, or point a webhook at an admin port on localhost. The OWASP SSRF Prevention Cheat Sheet covers this case, where the destinations cannot be allowlisted because they are not known in advance, and falls back to a deny list, while warning that deny lists are easier to bypass than allow lists.

import ipaddress
import socket
from urllib.parse import urlsplit

def resolve_public(url: str) -> list:
    """Resolve a customer's webhook URL; refuse it if any address is not public."""
    parts = urlsplit(url)
    if parts.scheme != "https" or not parts.hostname:
        raise ValueError("only https URLs are accepted")
    infos = socket.getaddrinfo(parts.hostname, parts.port or 443, proto=socket.IPPROTO_TCP)
    addrs = sorted({info[4][0] for info in infos})
    for a in addrs:
        ip = ipaddress.ip_address(a.split("%")[0])  # drop an IPv6 zone id
        # is_global is False for loopback, RFC 1918, link-local (cloud metadata
        # at 169.254.169.254), unique local IPv6 and other reserved ranges.
        if not ip.is_global or ip.is_multicast:
            raise ValueError(f"{parts.hostname} resolves to a non-public address")
    return addrs

# Connect to one of these checked addresses rather than resolving the name again,
# keep the hostname for the Host header and TLS, and never follow redirects.

• Check every resolved address, not the hostname. OWASP lists the ranges to block, including loopback, the RFC 1918 private ranges, link-local, unique local IPv6, multicast and the cloud metadata addresses.
• Close the DNS rebinding gap. OWASP warns that a second DNS lookup between the check and the connection can bypass it; connect to the address you checked, keeping the original hostname for the Host header, SNI and certificate checks.
• Turn off redirect following in the HTTP client. A public URL that redirects to an internal one would otherwise slip past the check, and webhooks treat 3xx as a failure anyway.
• Check again at delivery time, not only when the URL is saved: DNS can change after registration.
• Add a network layer. Standard Webhooks recommends sending through a filtering egress proxy that blocks internal addresses, naming Stripe’s open-source Smokescreen as an example, and running delivery workers in a subnet that cannot reach internal services.

The receiver’s side: idempotency and a fast 2xx

Your documentation should tell customers how to receive webhooks well, because most delivery problems are receiver problems. GitHub suggests putting deliveries on a queue and processing them asynchronously so the 2xx goes back at once; Stripe says to return a 2xx before any complex logic that might time out.

• Verify the signature on the raw body, then store the event and return 2xx. Do the work in a background job.
• Dedupe on the event id. Standard Webhooks says to use webhook-id as the idempotency key; Stripe says to log the event ids you have processed and skip the ones already logged. A unique constraint on the id, written in the same transaction as the work, is the reliable form (see idempotency and job queues).
• Do not rely on order. Stripe warns that its created timestamps are in seconds, so distinct events can share one, and tells receivers to fetch missing objects through the API when an event arrives early.
• Keep the secret out of source control and rotate it if it leaks.

Thin vs fat events

A fat (or full, or snapshot) event carries the whole changed object. A thin event carries the event type, an id and perhaps a few fields, and the receiver calls your API for the current state. Stripe offers both and its setup flow labels thin as recommended; the Standard Webhooks specification sets out the trade-offs and allows mixes, such as a thin event with one commonly needed field.

Fan-out, per-tenant fairness and delivery logs

One event can go to many endpoints, and one tenant can produce a flood of events. Fan out at the relay: one event becomes one delivery record per subscribed endpoint, each retried and logged on its own, so one endpoint’s failure never affects another’s copy.

• Cap in-flight requests per endpoint. An endpoint that takes the full timeout on every request can otherwise hold every worker, and its owner’s outage becomes everyone’s delay.
• Schedule fairly across tenants: round-robin or weighted queues per tenant, rather than one first-in, first-out queue where a tenant importing a million records goes first for an hour.
• Move endpoints that keep failing to a separate, slower lane so healthy endpoints keep their latency.
• Rate-limit per endpoint and honor 429 and retry-after; the token bucket in designing a rate limiter works on the sending side too.

The delivery log is what turns support tickets into self-service. For every attempt, keep the event id and type, the endpoint, the attempt number, the timestamp, the request headers and body as sent (with the signature, never the secret), the response status code, the latency, the first few kilobytes of the response body, the error class (timeout, DNS, TLS, refused) and the next attempt time. Stripe’s Event deliveries view shows each event as delivered, pending or failed, with the status code and the time of the next retry. Add a button to resend one event and another to send a test event, and set a retention period for the log.

BYD Tech builds web apps and APIs, webhooks and background jobs included: the code lives in the client’s GitHub from day one, and AI-assisted work is reviewed by an engineer.