Webhook Idempotency and Retries: 4 Receiver Handlers Tested

Why am I getting the same webhook twice, and how do I stop it? Providers like Stripe, Shopify and Svix deliver webhooks at least once, so duplicates are part of the design, and the fix is on the receiving end: make the handler idempotent by storing each event's ID under a UNIQUE constraint in the same database transaction as the side effect, and skip the event when that insert conflicts.

We tested that claim instead of repeating it. In a seeded simulation of 10,000 events on Node 24 and SQLite, a naive handler over-credited 354 events, the common check-then-insert handler still double-applied 65 events when two copies arrived at once, and the atomic UNIQUE insert applied every event exactly once. A version guard on top was the only design that left every account on its latest plan.

That leaves three decisions this post walks through: which key to store, how long to keep it (a 5-minute TTL let 247 to 325 duplicates through in our retention test), and how to handle events that arrive out of order. The script, versions and unedited output are at the end.

Webhook idempotency and retries: the short answer

A webhook handler is idempotent when processing the same event twice has the same effect as processing it once. You need that because every major provider retries: if your endpoint doesn't return a 2xx in time, the provider sends the event again, even when your code already ran. Retries and duplicates are normal operation, not a bug to report.

The checklist we'd ship:

1. Verify the signature on the raw body before you trust anything in the request.
2. Store the event durably, then return 2xx. Do the slow work from a queue.
3. Deduplicate with a UNIQUE insert on the provider's delivery or event ID, in the same transaction as the side effect.
4. Keep the keys longer than the provider can resend, which for Stripe means at least 30 days.
5. Guard order per object with a stored version, not with the event's timestamp.

Why providers send the same webhook twice

Providers resend because they can't tell the difference between "your server never got it" and "your server did the work but the response got lost." Both look like a timeout from the sender's side. Stripe's webhooks guide says endpoints "might occasionally receive the same event more than once," and Shopify's verify deliveries page says: "Shopify minimizes duplicate deliveries, but your app might receive the same webhook more than once, for example after a network timeout or a retry."

The usual sources of a second copy:

• A timeout after your side effect. Your handler credits the account, then takes six seconds to respond. Shopify gave up at five, so it retries an event you already applied.
• Retries after errors. A 500 from a deploy, a database blip or a bug triggers the provider's retry schedule.
• Manual resends. Someone clicks Resend in a dashboard days later. Stripe's Dashboard allows this for 15 days after the event is created, and its CLI for 30.
• Concurrent deliveries. A retry can land while the first attempt is still running, so two copies of one event are inside your handler at the same moment.
• Separate events for the same change. Stripe notes that "in some cases, two separate Event objects are generated and sent," and suggests identifying those duplicates by "the ID of the object in data.object along with the event.type."

This is a different problem from idempotency keys on outbound API calls, where you are the client retrying a request to someone else's API. We measured that side, including duplicate invoices from retries without a key, in our monolith vs microservices benchmark.

We tested four receiver handlers under duplicate, concurrent and out-of-order delivery

The atomic UNIQUE insert was the only plain dedupe design with zero duplicate side effects, and only the version guard kept every account on its latest plan. Here is how we got there.

Method. This is a simulation, not production data. A seeded generator (seed 20261023) created 10,000 synthetic events across 500 accounts: 5,935 "add 100 credits" events and 4,065 "set plan" events, each plan update carrying a per-account version and a created timestamp in whole seconds. Four faults were drawn with chances picked by us rather than measured anywhere. Three are drawn once per event, for its first delivery; the lost response is drawn again on every attempt (the script's output header calls them all "chances per delivery"):

A fifth fault was built into the events: each plan update had a 0.05 chance of being followed by a second update for the same account in the same second, which produced 193 same-second pairs. Every handler saw the same fault draws, because each draw is keyed by event ID and attempt number rather than by processing order.

The receivers ran on Node v24.20.0 with the built-in node:sqlite module (SQLite 3.53.4), which the Node 24 docs label "Stability: 1.2 - Release candidate" (since v24.15.0). Node printed no experimental warning on our runs. DatabaseSync is synchronous, so we modelled concurrency the way it happens in a Node server: every handler awaits simulated I/O, and two concurrent copies interleave at that await. We ran the script three times on an Apple M1 Max (macOS 27.0.1) on 2026-10-23; all three outputs were byte-for-byte identical.

The handlers:

• (a) Naive: apply the side effect on every delivery.
• (b) Check-then-insert: SELECT the event ID, await I/O, apply the side effect, then INSERT the ID into a table with a primary key. It is a common tutorial pattern.
• (c) Atomic insert: in one transaction, INSERT ... ON CONFLICT DO NOTHING into processed_events, and apply the side effect only if a row was inserted. No await inside the transaction.
• (d) Atomic insert plus an ordering guard, in three variants: apply a plan update only if created >= the stored value, only if created > it, or only if its version is greater than the stored version.

Results

"Stale writes" counts the times an older plan overwrote a newer one at any point in the run; "accounts ending on an old plan" checks only the final state. No handler lost a credit event in this run, because every event was delivered at least once and nothing crashed. The crash test further down is where events go missing.

What the table says:

• The naive handler over-credited by 36,800, about 6.2% of the expected total, from 368 extra applies on 354 events.
• Check-then-insert removed most duplicates but not the concurrent ones. Both copies passed the SELECT before either wrote its row. The second copy then hit the primary key, but only after it had already granted the credit, so it returned a 500 and triggered a retry for an event it had just applied. That is where the 106 errors and the 65 double-applied events come from.
• The atomic insert fixed duplicates and nothing else. It still let 23 shuffled updates overwrite newer plans, because deduplication says nothing about order.
• Ordering by created traded one failure for another. With >=, 17 same-second updates overwrote a newer plan. With >, nothing was overwritten, but newer updates that shared a second with the stored one were rejected, and 19 accounts ended the run on an older plan.

Traced example: one credit, one tie

Follow evt_000132, "add 100 credits to acct_0243," the first credit event in the stream that arrived as a concurrent pair. The naive handler applied it twice and returned 200 both times. Check-then-insert also applied it twice: attempt 2 got a 500 from the primary key, the sender retried, and attempt 3 found the row and returned 200, by which point the account had been credited twice. The atomic insert applied it once; the second copy's insert conflicted, so it committed nothing and returned 200.

Then the ordering case. On acct_0388, evt_000090 set plan_v1 and evt_000091 set plan_v2, both with created 1790000039, and v2 arrived first. After both deliveries, handlers (a), (b), (c) and the created >= variant held plan_v1, the older value. The created > variant and the version guard held plan_v2. The created > variant only got this one right because the newer event happened to arrive first; it is the variant that left 19 accounts stale.

What this simulation does not show

• It is one process with one SQLite connection. Concurrency is modelled by interleaving at an await, not by multiple workers or real network calls. It shows which designs can fail and how, not how often your production traffic will trigger it. The fault chances are ours, not measured rates.
• In Postgres the same design is INSERT ... ON CONFLICT DO NOTHING inside the side effect's transaction. Across several workers that relies on the unique index, which our single-process run doesn't exercise.
• When the side effect leaves your database, such as sending an email or calling a third-party API, a database transaction can't make it exactly-once. You need the downstream API's own idempotency key or an outbox table that a separate worker drains.
• We don't publish production webhook data from clients or from our own systems.

Verify the signature on the raw body first

Verify before you deduplicate, so a forged request never writes a key. The signature is an HMAC over the exact bytes the provider sent, so compute it over the raw body; parsing the JSON and serializing it again changes whitespace and key order and breaks the match. Stripe's docs say "Stripe requires the raw body of the request to perform signature verification," and Shopify's say "HMAC verification requires the raw request body."

What gets signed differs by provider:

• The Standard Webhooks spec sends webhook-id, webhook-timestamp and webhook-signature headers, and "the content to be signed is therefore: msg_id.timestamp.payload." Svix uses the same scheme with svix- header names.
• Stripe signs the timestamp, a . and the payload, and its libraries "have a default tolerance of 5 minutes between the timestamp and the current time."
• Stripe also says: "If Stripe retries an event ... then we generate a new signature and timestamp for the new delivery attempt." So never deduplicate on the signature or the signed timestamp; two copies of one event won't match. Use the event ID.

Compare signatures in constant time (crypto.timingSafeEqual in Node, after checking the lengths match), and reject timestamps outside your tolerance to block replays.

Acknowledge fast, then queue the work

Return a 2xx as soon as the event is stored durably, and process it afterwards. Provider timeouts are short:

• Shopify: "Shopify has a one-second connection timeout and a five-second timeout for the entire request."
• GitHub: "Your server should respond with a 2XX response within 10 seconds of receiving a webhook delivery" (best practices).
• Standard Webhooks: "A recommended request timeout value for webhooks is somewhere between 15 and 30s."
• Svix: a success response must arrive "within a reasonable time-frame (30s with Svix)" (retries).
• Stripe gives no number but says your endpoint "must quickly return a successful status code (2xx) before any complex logic that could cause a timeout."

The important word is durably. Returning 200 before anything is saved and then crashing loses the event for good, because the provider marks it delivered. The safe order is: verify, insert into an inbox table with a UNIQUE key on the delivery ID, return 200 (also when the insert conflicts, since that copy is already stored), then let a worker apply the side effect. The worker's processing must be idempotent too: mark the inbox row processed in the same transaction as the side effect, so a worker crash leaves the row unprocessed for the next attempt.

An illustrative Express receiver and worker for that design (not run as part of this test; the simulation is the tested part):

CREATE TABLE webhook_inbox (
  id           bigserial PRIMARY KEY,
  source       text NOT NULL,
  webhook_id   text NOT NULL,
  payload      jsonb NOT NULL,
  received_at  timestamptz NOT NULL DEFAULT now(),
  processed_at timestamptz,
  attempts     int NOT NULL DEFAULT 0,
  last_error   text,
  UNIQUE (source, webhook_id)
);

import express from "express";
import crypto from "node:crypto";
import pg from "pg";

const pool = new pg.Pool();
// Standard Webhooks secrets look like "whsec_<base64>"
const secret = Buffer.from(process.env.WEBHOOK_SECRET.replace(/^whsec_/, ""), "base64");
const TOLERANCE_S = 5 * 60;
const app = express();

function verify(req) {
  const id = req.get("webhook-id"); // req.get is case-insensitive
  const ts = req.get("webhook-timestamp");
  const header = req.get("webhook-signature");
  if (!id || !ts || !header || !Buffer.isBuffer(req.body)) return null;
  const t = Number(ts);
  if (!Number.isFinite(t) || Math.abs(Date.now() / 1000 - t) > TOLERANCE_S) return null;
  const expected = crypto.createHmac("sha256", secret).update(`${id}.${ts}.`).update(req.body).digest();
  for (const part of header.split(" ")) {
    const [version, value] = part.split(",");
    if (version !== "v1" || !value) continue;
    const got = Buffer.from(value, "base64");
    if (got.length === expected.length && crypto.timingSafeEqual(got, expected)) return id;
  }
  return null;
}

app.post("/webhooks/acme", express.raw({ type: "application/json", limit: "1mb" }), async (req, res) => {
  const id = verify(req);
  if (!id) return res.sendStatus(400);
  try {
    await pool.query(
      `INSERT INTO webhook_inbox (source, webhook_id, payload)
       VALUES ('acme', $1, $2)
       ON CONFLICT (source, webhook_id) DO NOTHING`,
      [id, req.body.toString("utf8")]
    );
    res.sendStatus(200); // stored now or stored before: either way, ack
  } catch (err) {
    console.error("webhook inbox insert failed", err);
    res.sendStatus(500); // not stored: let the sender retry
  }
});

// Worker: one row at a time, side effect and processed_at in one transaction.
export async function processNext(applyEvent) {
  const client = await pool.connect();
  let rowId = null;
  try {
    await client.query("BEGIN");
    const { rows } = await client.query(
      `SELECT id, payload FROM webhook_inbox
       WHERE processed_at IS NULL AND attempts < 5
       ORDER BY received_at
       FOR UPDATE SKIP LOCKED
       LIMIT 1`
    );
    if (rows.length === 0) { await client.query("COMMIT"); return false; }
    rowId = rows[0].id;
    await applyEvent(client, rows[0].payload); // credits, plan with version guard, etc.
    await client.query("UPDATE webhook_inbox SET processed_at = now() WHERE id = $1", [rowId]);
    await client.query("COMMIT");
    return true;
  } catch (err) {
    await client.query("ROLLBACK");
    if (rowId !== null) {
      await pool.query("UPDATE webhook_inbox SET attempts = attempts + 1, last_error = $2 WHERE id = $1", [rowId, String(err)]);
    }
    throw err;
  } finally {
    client.release();
  }
}

For multi-shop or multi-tenant apps, add the shop or tenant to the unique key, the same tenant scoping we cover in multi-tenant SaaS architecture.

Webhook deduplication: choosing the key, the store and how long to keep it

Deduplicate on the ID the provider keeps the same across retries, in a database table with a UNIQUE constraint, and keep the rows for longer than the provider can resend. Each part has a trap.

The key, per provider

• Stripe: the event's id. Stripe's guide says to "track event IDs to identify duplicate deliveries," and to catch the rare case of two separate Event objects, also compare data.object ID plus event.type.
• Shopify: X-Shopify-Webhook-Id, not X-Shopify-Event-Id. With more than one subscription to a topic, "each has a different X-Shopify-Webhook-Id but shares the same X-Shopify-Event-Id." Shopify says to use the webhook ID "to deduplicate individual deliveries" and the event ID "to correlate deliveries that originated from the same merchant action." Deduplicating on the event ID silently drops your second subscription's deliveries.
• GitHub: X-GitHub-Delivery. Its docs say "if you request a redelivery, the X-GitHub-Delivery header will be the same as in the original delivery."
• Standard Webhooks and Svix: webhook-id (or svix-id), which Svix's manual verification page says "will be the same when the same webhook is being resent."

The store: a UNIQUE insert, committed with the side effect

Check-then-insert looks correct and reads well in a code review, which is why it ships. Our run shows why it isn't enough: under concurrent redelivery it double-applied 65 events, because the check and the write are two steps with a gap between them. A UNIQUE insert closes the gap, since the database decides which copy wins.

The other half is committing the key and the side effect together. We tested what happens when they are separate: 2,000 credit events, file-backed SQLite, and a 0.01 chance per delivery that the process dies after the dedupe insert and before the side effect (we close the database connection without committing, then reopen it):

In the split design every crash lost its event permanently: the key was already committed, so the retry looked like a duplicate and was skipped. In the single transaction, SQLite discarded the uncommitted insert when the connection closed, and the retry processed the event normally. The same applies to the inbox pattern above: the inbox row is the key, and processed_at has to commit with the side effect.

A Redis key with a TTL is useful as a fast first filter, but it isn't a guarantee: it lives apart from your database transaction and it expires. Treat it as a cache in front of the UNIQUE constraint, not as a replacement.

How long to keep the keys

Keep keys for at least the provider's full retry window plus its manual resend window. For Stripe that's 30 days, because a resend from the Stripe CLI "works for up to 30 days after the event creation." For Shopify, retries run "8 times over the next 4 hours." For GitHub, manual redelivery covers "deliveries that occurred in the past 3 days."

The Standard Webhooks spec shows why this needs saying. Its idempotency advice suggests you "save the IDs in redis for 5 minutes," while the same document's example retry schedule ends at 75:35:05, that is 75 hours and 35 minutes after the first attempt. A key kept for five minutes is long gone before most of those retries arrive.

We replayed that mismatch. For 10,000 events per schedule, each attempt that reached the handler had a 0.05 chance of its response getting lost after processing, and each later attempt had a 0.5 chance of never reaching the receiver (all chosen values). The 3-day row also had a 0.005 chance per event of a manual resend 1 to 15 days later:

The third row stands in for a Stripe-like window. Stripe doesn't publish its exact intervals, so those are ours. Seven days covered every automatic retry but still let 28 manual resends through; only the 30-day retention caught them all. Storage is rarely the constraint here: one row per event with a short text key, deleted by a nightly job after 30 or 45 days.

Out-of-order webhook events: don't trust timestamps

Apply an update only if it's newer than the state you've stored for that object, and decide "newer" with a version or the resource's own last-modified value, not with arrival order or the event's creation second. No provider in this post promises order:

• Stripe: "Stripe doesn't guarantee the delivery of events in the order that they're generated." It adds that "snapshot events record created in seconds, so distinct events can share a timestamp," and "don't use created to determine event order or whether you've already processed an event."
• Shopify's webhooks overview: "Shopify doesn't guarantee ordering within a topic, or across different topics for the same resource."

Shopify does recommend using X-Shopify-Triggered-At or the payload's updated_at "to organize webhooks." That's a reasonable signal, with one caveat our run makes concrete: whichever timestamp you use, decide what happens on a tie. Ordering by created >= let 17 same-second updates overwrite a newer plan, and created > left 19 accounts on an older plan. Neither is safe when two changes land in the same second.

What we'd do, in order of preference:

1. A version from the source. If the payload carries a monotonically increasing version or sequence for the object, store the last one applied and run UPDATE ... SET plan = $2, plan_version = $3 WHERE id = $1 AND plan_version < $3. If no row is updated, the event was stale: record it as processed and move on. This variant had zero stale writes and zero stale accounts.
2. A last-modified value compared against stored state. Use the resource's own updated_at with a strict comparison, and treat a tie as "unknown," not as newer or older.
3. Re-fetch on doubt. On a tie, or for events where order matters a lot (status changes, cancellations), call the provider's API for the current object and store that. Stripe's docs point the same way for missing objects: "You can also use the API to retrieve any missing objects." Using the re-fetch for ties is our recommendation, not a provider rule.

Payload and schema changes across API versions are a separate concern, which we cover in API versioning best practices.

Poison events and a dead-letter queue

An event that fails every time must not block the queue or keep returning 5xx to the provider. Return 2xx once the event is stored, then let the worker retry it a bounded number of times (the attempts < 5 filter in the worker above), and leave rows that hit the limit in place as your dead-letter set. Alert on that set, fix the bug, reset attempts and let the worker replay them.

Keeping failures on your side matters more with some providers than others. Shopify's docs say "after 8 consecutive failures, the subscription is automatically deleted if it was configured using the Admin API," and the troubleshooting page puts it plainly: "if failures persist, the subscription is removed." A handler that returns 500 on one malformed payload can lose the whole subscription, along with every healthy event behind it. For Shopify-specific peak-season failure handling, see our Shopify Black Friday checklist.

Webhook retry best practices for senders

If you send webhooks, use exponential backoff with jitter, a timeout in the 15 to 30 second range, a retry window measured in days, a documented rule for disabling dead endpoints, and a replay button for your customers. Here is what the five providers in this post document, each page read on 2026-10-23:

The GitHub row is the easiest one to get wrong. Don't build on the assumption that GitHub retries: if your endpoint is down, nothing comes back on its own. GitHub's redelivery page covers the manual and API routes, and its best practices say "if your server goes down, you should redeliver missed webhooks once your server is back up." That makes the recovery job your responsibility.

Jitter. We simulated 1,000 deliveries that failed at the same moment against a receiver that came back after 300 seconds, retrying with exponential backoff from 1 second. Without jitter, all 1,000 retries arrived in the same second at t=511 s, a synchronized spike at exactly the moment the receiver was recovering. With full jitter (a random delay between zero and the backoff value) the peak after recovery was 11 per second. The trade-off showed up too: the last jittered delivery landed at t=1781 s, and the run made 10,790 attempts instead of 10,000. Spread is worth a slower tail. The same backoff and Retry-After reasoning applies to API clients, which we cover in SaaS API rate limiting.

The rest of the sender's list:

• Disable endpoints by rule and tell the customer. Svix disables after 5 days of failures, Standard Webhooks treats 410 Gone as a request to stop, and Shopify removes the subscription. Whatever you choose, email the owner before and after.
• Give customers replay. A "resend" button and a replay-since-timestamp API turn an outage on their side into a recoverable one.
• Use a transactional outbox. Write the event row in the same transaction as the business change and send from that table, so you never emit an event for a write that rolled back. We walk through the same pattern for usage events in usage-based billing for SaaS.
• Keep IDs stable across retries, and put a version or sequence per object in the payload. That one field lets receivers solve ordering without guessing.

Testing checklist before you go live

Run these against staging before the first real event arrives:

1. Send the same event twice in a row, and confirm the side effect happened once.
2. Send two copies at the same time (two parallel curl calls are enough), and confirm it still happened once.
3. Shuffle a batch of updates for one object, and confirm the final state is the latest version.
4. Kill the worker between the inbox insert and the side effect, restart it, and confirm the event is applied once, not zero times.
5. Replay a day-old event, and confirm the key is still there to catch it.
6. Rotate the signing secret. Stripe lets you keep the old secret active for up to 24 hours, so test the overlap.
7. Check the retention job, and that it keeps keys at least as long as the provider can resend.
8. Return 500 from one event on purpose, and confirm it lands in the dead-letter set while the events behind it keep flowing.

If you're building a webhook-heavy integration or fixing one that double-charges, this receiver design and its tests are part of our API and SaaS development work, and for Shopify apps, of our Shopify development work.

The full simulation script and output

Save the script as webhook-sim.mjs and run node webhook-sim.mjs. It needs Node 24 or later (for node:sqlite) and nothing else: no npm install, no network. It takes a few seconds. The crash test writes a temporary SQLite file in your OS temp directory and deletes it afterwards. Fault chances are constants at the top; change them and the seed to see how the counts move.

Run metadata: seed 20261023, Node v24.20.0, SQLite 3.53.4 (bundled with Node), Apple M1 Max (10 cores, 64 GB RAM), macOS 27.0.1, run three times on 2026-10-23 with byte-for-byte identical output.

// webhook-sim.mjs: a seeded simulation of webhook receivers under at-least-once delivery.
// Node 24+, no dependencies. Run: node webhook-sim.mjs
// Everything here is synthetic: event names, accounts and fault rates are chosen, not measured.
import { DatabaseSync } from "node:sqlite";
import os from "node:os";
import fs from "node:fs";
import path from "node:path";

const SEED = 20261023;
const N_EVENTS = 10000;
const N_ACCOUNTS = 500;
const P_CREDIT = 0.6;       // chance an event is "add 100 credits" (else "set plan")
const P_TIE = 0.05;         // chance a plan update gets a second update for the same account in the same second
const P_DUPLICATE = 0.03;   // fault 1: chance a delivery is sent again later
const P_ACK_LOST = 0.02;    // fault 2: chance a delivery's 200 response is lost after the handler committed
const P_CONCURRENT = 0.01;  // fault 3: chance the first delivery arrives twice at the same moment
const P_SHUFFLE = 0.10;     // fault 4: chance a delivery is moved up to 20 positions
const SHUFFLE_WINDOW = 20;

// ---------- deterministic randomness ----------
function mulberry32(a) {
  return function () {
    a |= 0; a = (a + 0x6d2b79f5) | 0;
    let t = Math.imul(a ^ (a >>> 15), 1 | a);
    t = (t + Math.imul(t ^ (t >>> 7), 61 | t)) ^ t;
    return ((t ^ (t >>> 14)) >>> 0) / 4294967296;
  };
}
// A draw keyed by (event, attempt, purpose), so every handler sees the same faults
// no matter in which order it processes deliveries.
function keyed(...parts) {
  let h = SEED >>> 0;
  for (const ch of parts.join("|")) h = Math.imul(h ^ ch.charCodeAt(0), 2654435761) >>> 0;
  return mulberry32(h)();
}

// ---------- logical events ----------
function makeEvents() {
  const rnd = mulberry32(SEED);
  const events = [];
  const planVersion = new Array(N_ACCOUNTS).fill(0);
  let t = 1790000000; // synthetic epoch seconds
  const push = (e) => { e.id = "evt_" + String(events.length).padStart(6, "0"); e.idx = events.length; events.push(e); };
  while (events.length < N_EVENTS) {
    if (rnd() < 0.5) t += 1;
    const acct = Math.floor(rnd() * N_ACCOUNTS);
    if (rnd() < P_CREDIT) {
      push({ type: "credit", acct, amount: 100, created: t });
    } else {
      push({ type: "plan", acct, version: ++planVersion[acct], created: t });
      if (rnd() < P_TIE && events.length < N_EVENTS) {
        push({ type: "plan", acct, version: ++planVersion[acct], created: t, tie: true });
      }
    }
  }
  return { events, planVersion };
}

// ---------- delivery stream ----------
function makeGroups(events) {
  const groups = [];
  for (const e of events) {
    let at = e.idx;
    if (keyed(e.id, "shuffle") < P_SHUFFLE) at += (keyed(e.id, "shift") * 2 - 1) * SHUFFLE_WINDOW;
    const copies = keyed(e.id, "concurrent") < P_CONCURRENT ? 2 : 1;
    groups.push({ at, event: e, copies, kind: copies === 2 ? "first+concurrent" : "first" });
    if (keyed(e.id, "dup") < P_DUPLICATE) {
      groups.push({ at: e.idx + 1 + keyed(e.id, "dupAt") * 200, event: e, copies: 1, kind: "duplicate" });
    }
  }
  return groups.sort((a, b) => a.at - b.at || a.event.idx - b.event.idx);
}

// ---------- receivers ----------
const io = () => new Promise((r) => setImmediate(r)); // simulated I/O (parse, verify, network)

function freshDb(file = ":memory:") {
  const db = new DatabaseSync(file);
  db.exec(`
    CREATE TABLE IF NOT EXISTS accounts (id INTEGER PRIMARY KEY, credits INTEGER NOT NULL DEFAULT 0,
      plan TEXT, plan_version INTEGER NOT NULL DEFAULT 0, plan_created INTEGER NOT NULL DEFAULT 0);
    CREATE TABLE IF NOT EXISTS processed_events (event_id TEXT PRIMARY KEY);
    CREATE TABLE IF NOT EXISTS applied (event_id TEXT NOT NULL);`);
  if (db.prepare("SELECT count(*) n FROM accounts").get().n === 0) {
    const ins = db.prepare("INSERT INTO accounts (id) VALUES (?)");
    db.exec("BEGIN"); for (let i = 0; i < N_ACCOUNTS; i++) ins.run(i); db.exec("COMMIT");
  }
  return db;
}

function prep(db) {
  return {
    credit: db.prepare("UPDATE accounts SET credits = credits + ? WHERE id = ?"),
    setPlan: db.prepare("UPDATE accounts SET plan = ?, plan_version = ?, plan_created = ? WHERE id = ?"),
    getAcct: db.prepare("SELECT plan_version, plan_created FROM accounts WHERE id = ?"),
    applied: db.prepare("INSERT INTO applied (event_id) VALUES (?)"),
    seen: db.prepare("SELECT 1 FROM processed_events WHERE event_id = ?"),
    mark: db.prepare("INSERT INTO processed_events (event_id) VALUES (?)"),
    claim: db.prepare("INSERT INTO processed_events (event_id) VALUES (?) ON CONFLICT DO NOTHING"),
    staleWrites: 0,
  };
}

function apply(s, e, guard) {
  if (e.type === "credit") { s.credit.run(e.amount, e.acct); s.applied.run(e.id); return; }
  const cur = s.getAcct.get(e.acct);
  if (guard === "version" || guard === "created>=" || guard === "created>") {
    const ok = guard === "version" ? e.version > cur.plan_version
      : guard === "created>=" ? e.created >= cur.plan_created
      : e.created > cur.plan_created;
    if (!ok) return; // stale: recorded as processed, not applied
  }
  if (e.version < cur.plan_version) s.staleWrites++; // an older plan overwrote a newer one
  s.setPlan.run("plan_v" + e.version, e.version, e.created, e.acct);
  s.applied.run(e.id);
}

const handlers = {
  "a naive": async (db, s, e) => { await io(); apply(s, e); return 200; },
  "b check-then-insert": async (db, s, e) => {
    if (s.seen.get(e.id)) return 200;
    await io();
    apply(s, e);
    try { s.mark.run(e.id); } catch { return 500; }
    return 200;
  },
  "c atomic insert": async (db, s, e) => atomic(db, s, e, null),
  "d order by created>=": async (db, s, e) => atomic(db, s, e, "created>="),
  "d order by created>": async (db, s, e) => atomic(db, s, e, "created>"),
  "d version guard": async (db, s, e) => atomic(db, s, e, "version"),
};

async function atomic(db, s, e, guard) {
  await io(); // the await happens before the transaction, never inside it
  db.exec("BEGIN IMMEDIATE");
  try {
    if (s.claim.run(e.id).changes === 0) { db.exec("COMMIT"); return 200; } // duplicate: skip and ack
    apply(s, e, guard);
    db.exec("COMMIT");
    return 200;
  } catch (err) { db.exec("ROLLBACK"); throw err; }
}

async function runHandler(name, events, groups, latestPlan, trace, tieTrace) {
  const db = freshDb();
  const s = prep(db);
  const h = handlers[name];
  const queue = groups.map((g) => ({ ...g }));
  const attempts = new Map();
  let deliveries = 0, errors = 0;
  const log = [];
  let tieAfter = null;
  const enqueue = (item) => {
    let i = queue.findIndex((q) => q.at > item.at);
    if (i === -1) i = queue.length;
    queue.splice(i, 0, item);
  };
  while (queue.length) {
    const g = queue.shift();
    const e = g.event;
    const runs = [];
    for (let c = 0; c < g.copies; c++) {
      const n = (attempts.get(e.id) || 0) + 1;
      attempts.set(e.id, n);
      runs.push(h(db, s, e).then((status) => ({ status, n })));
    }
    for (const { status, n } of await Promise.all(runs)) {
      deliveries++;
      if (e.id === trace) log.push(`${g.kind} attempt ${n}: ${status}`);
      if (e.id === tieTrace && n === 1) tieAfter = s.getAcct.get(e.acct).plan_version;
      if (status >= 500) {
        errors++;
        enqueue({ at: g.at + 5 + keyed(e.id, n, "retryAt") * 200, event: e, copies: 1, kind: "retry-after-500" });
      } else if (keyed(e.id, n, "ackLost") < P_ACK_LOST) {
        enqueue({ at: g.at + 5 + keyed(e.id, n, "retryAt") * 200, event: e, copies: 1, kind: "retry-after-timeout" });
        if (e.id === trace) log[log.length - 1] += " (response lost, sender retries)";
      }
    }
  }
  const credits = events.filter((e) => e.type === "credit");
  const appliedCounts = new Map(db.prepare("SELECT event_id, count(*) c FROM applied GROUP BY event_id").all().map((r) => [r.event_id, r.c]));
  let dupEvents = 0, dupApplies = 0, lost = 0;
  for (const e of credits) {
    const c = appliedCounts.get(e.id) || 0;
    if (c === 0) lost++;
    if (c > 1) { dupEvents++; dupApplies += c - 1; }
  }
  let stale = 0;
  for (const r of db.prepare("SELECT id, plan_version FROM accounts").all()) {
    if (latestPlan[r.id] > 0 && r.plan_version !== latestPlan[r.id]) stale++;
  }
  const ledger = db.prepare("SELECT sum(credits) s FROM accounts").get().s;
  const expected = credits.length * 100;
  const tracedApplied = appliedCounts.get(trace) || 0;
  db.close();
  return { name, deliveries, errors, dupEvents, dupApplies, lost, stale, staleWrites: s.staleWrites, ledger, expected, log, tracedApplied, tieAfter };
}

// ---------- retention test: dedupe keys with a TTL vs a provider retry schedule ----------
function retention() {
  const H = 3600, D = 86400;
  const cum = (delays) => { let t = 0; return delays.map((d) => (t += d)); };
  const schedules = {
    "Svix published schedule": cum([0, 5, 300, 1800, 2 * H, 5 * H, 10 * H, 10 * H]),
    "Standard Webhooks example": cum([0, 5, 300, 1800, 2 * H, 5 * H, 10 * H, 14 * H, 20 * H, 24 * H]),
    "3-day window, illustrative intervals": [0, 60, 360, 2160, 5760, 12960, 27360, 56160, 113760, 172800, 259200],
  };
  const ttls = [["5 min", 300], ["24 h", D], ["7 days", 7 * D], ["30 days", 30 * D]];
  const P_LOST = 0.05, P_UNREACHABLE = 0.5, P_MANUAL = 0.005;
  const rows = [];
  for (const [sname, offsets] of Object.entries(schedules)) {
    const manual = sname.startsWith("3-day");
    for (const [tname, ttl] of ttls) {
      let applies = 0;
      for (let i = 0; i < N_EVENTS; i++) {
        let expires = -1, done = false;
        const visit = (t, n) => {
          if (t >= expires) { applies++; expires = t + ttl; }
          return keyed("ret", sname, i, n, "lost") >= P_LOST; // true when the 200 reaches the sender
        };
        for (let n = 0; n < offsets.length && !done; n++) {
          if (n > 0 && keyed("ret", sname, i, n, "down") < P_UNREACHABLE) continue; // never reached us
          done = visit(offsets[n], n);
        }
        if (manual && keyed("ret", sname, i, "manual") < P_MANUAL) {
          visit(D + keyed("ret", sname, i, "manualAt") * 14 * D, "manual");
        }
      }
      rows.push([sname, tname, applies - N_EVENTS]);
    }
    rows.push([sname, "last retry at", fmt(offsets[offsets.length - 1])]);
  }
  return rows;
}
function fmt(sec) {
  const h = Math.floor(sec / 3600), m = Math.floor((sec % 3600) / 60), s = sec % 60;
  return `${h}:${String(m).padStart(2, "0")}:${String(s).padStart(2, "0")}`;
}

// ---------- crash test: process dies after the dedupe insert, before the side effect ----------
function crashTest(design) {
  const dir = fs.mkdtempSync(path.join(os.tmpdir(), "whsim-"));
  const file = path.join(dir, "crash.db");
  let db = freshDb(file), s = prep(db);
  const N = 2000, P_CRASH = 0.01;
  let crashes = 0;
  const queue = [];
  for (let i = 0; i < N; i++) queue.push({ id: "evt_c" + String(i).padStart(5, "0"), acct: i % N_ACCOUNTS, n: 1 });
  while (queue.length) {
    const d = queue.shift();
    const crash = keyed("crash", d.id, d.n) < P_CRASH;
    if (design === "split") {
      if (s.seen.get(d.id)) continue;            // already "processed": ack
      s.mark.run(d.id);                           // dedupe row commits on its own
      if (crash) { crashes++; db.close(); db = freshDb(file); s = prep(db); queue.push({ ...d, n: d.n + 1 }); continue; }
      s.credit.run(100, d.acct); s.applied.run(d.id);
    } else {
      db.exec("BEGIN IMMEDIATE");
      if (s.claim.run(d.id).changes === 0) { db.exec("COMMIT"); continue; }
      if (crash) { crashes++; db.close(); db = freshDb(file); s = prep(db); queue.push({ ...d, n: d.n + 1 }); continue; } // no COMMIT: SQLite discards the open transaction
      s.credit.run(100, d.acct); s.applied.run(d.id);
      db.exec("COMMIT");
    }
  }
  const applied = db.prepare("SELECT count(DISTINCT event_id) c FROM applied").get().c;
  const keys = db.prepare("SELECT count(*) c FROM processed_events").get().c;
  const ledger = db.prepare("SELECT sum(credits) s FROM accounts").get().s;
  db.close();
  fs.rmSync(dir, { recursive: true, force: true });
  return { design, N, crashes, keys, applied, lost: N - applied, ledger, expected: N * 100 };
}

// ---------- sender side: 1,000 failed deliveries retrying, with and without jitter ----------
function jitter() {
  const N = 1000, RECOVER = 300, BASE = 1, CAP = 3600;
  const out = [];
  for (const mode of ["exponential, no jitter", "exponential, full jitter"]) {
    const rnd = mulberry32(SEED + (mode.includes("full") ? 1 : 0));
    const buckets = new Map();
    let attempts = 0, lastOk = 0;
    for (let i = 0; i < N; i++) {
      let t = 0, k = 0;
      while (true) {
        attempts++;
        const sec = Math.floor(t);
        buckets.set(sec, (buckets.get(sec) || 0) + 1);
        if (t >= RECOVER) { lastOk = Math.max(lastOk, t); break; }
        const d = Math.min(CAP, BASE * 2 ** k++);
        t += mode.includes("full") ? rnd() * d : d;
      }
    }
    let peakAfter = 0;
    for (const [sec, n] of buckets) if (sec >= RECOVER) peakAfter = Math.max(peakAfter, n);
    out.push({ mode, attempts, peakAfter, lastOk: Math.round(lastOk) });
  }
  return out;
}

// ---------- main ----------
const pad = (v, n) => String(v).padEnd(n);
const lpad = (v, n) => String(v).padStart(n);
const probe = new DatabaseSync(":memory:");
console.log("webhook-sim (simulation, synthetic data)");
console.log("seed:", SEED);
console.log("node:", process.version);
console.log("sqlite:", probe.prepare("SELECT sqlite_version() v").get().v);
console.log("machine:", os.cpus()[0].model, os.platform(), os.arch());
console.log("run date:", new Date().toISOString().slice(0, 10));
probe.close();

const { events, planVersion } = makeEvents();
const groups = makeGroups(events);
const ties = events.filter((e) => e.tie).length;
console.log(`events: ${events.length} (${events.filter((e) => e.type === "credit").length} credit, ${events.filter((e) => e.type === "plan").length} plan), accounts: ${N_ACCOUNTS}, same-second plan pairs: ${ties}`);
console.log(`chances per delivery: duplicate ${P_DUPLICATE}, response lost after commit ${P_ACK_LOST}, concurrent redelivery ${P_CONCURRENT}, shuffled ${P_SHUFFLE} (window ${SHUFFLE_WINDOW})`);

// Traced event: first credit event with a concurrent pair, else the credit event with the most scheduled deliveries.
const perEvent = new Map();
for (const g of groups) perEvent.set(g.event.id, (perEvent.get(g.event.id) || 0) + g.copies);
const concurrentCredit = groups.find((g) => g.copies === 2 && g.event.type === "credit");
const trace = concurrentCredit ? concurrentCredit.event : events.filter((e) => e.type === "credit").sort((a, b) => perEvent.get(b.id) - perEvent.get(a.id))[0];
const latestPlan = planVersion;

// Traced tie: the first same-second pair whose newer update arrived first.
const pos = new Map();
groups.forEach((g, i) => { if (!pos.has(g.event.id)) pos.set(g.event.id, i); });
const reversed = events.filter((e) => e.tie && pos.get(e.id) < pos.get(events[e.idx - 1].id));
const tieTrace = reversed.length ? events[reversed[0].idx - 1].id : null;

const results = [];
for (const name of Object.keys(handlers)) results.push(await runHandler(name, events, groups, latestPlan, trace.id, tieTrace));

console.log("\n== handlers (simulation)");
console.log(pad("handler", 22) + lpad("deliveries", 11) + lpad("5xx", 6) + lpad("dup events", 11) + lpad("extra applies", 14) + lpad("lost", 6) + lpad("stale writes", 13) + lpad("stale accts", 12) + lpad("ledger", 10) + lpad("expected", 10));
for (const r of results) {
  console.log(pad(r.name, 22) + lpad(r.deliveries, 11) + lpad(r.errors, 6) + lpad(r.dupEvents, 11) + lpad(r.dupApplies, 14) + lpad(r.lost, 6) + lpad(r.staleWrites, 13) + lpad(r.stale, 12) + lpad(r.ledger, 10) + lpad(r.expected, 10));
}

console.log(`\n== traced event ${trace.id}: add ${trace.amount} credits to acct_${String(trace.acct).padStart(4, "0")}`);
for (const r of results.slice(0, 3)) {
  console.log(`${r.name}: applied ${r.tracedApplied}x`);
  for (const l of r.log) console.log("  " + l);
}

console.log(`\n== same-second pairs delivered newer-first: ${reversed.length} of ${ties}`);
if (reversed.length) {
  const e = reversed[0], older = events[e.idx - 1];
  console.log(`example acct_${String(e.acct).padStart(4, "0")}: ${older.id} sets plan_v${older.version}, ${e.id} sets plan_v${e.version}, both created ${e.created}; arrival order ${e.id} then ${older.id}`);
  for (const r of results) console.log(`  ${r.name}: plan_v${r.tieAfter} after both arrived`);
}

console.log("\n== retention: duplicate applies that got past the dedupe store (simulation)");
console.log("chances: response lost after processing 0.05 per attempt, later attempt unreachable 0.5, manual resend 0.005 (3-day row only, 1 to 15 days later)");
for (const [s, t, n] of retention()) console.log(pad(s, 38) + pad(t, 15) + lpad(n, 8));

console.log("\n== crash after dedupe insert, before side effect (simulation, file-backed SQLite, chance 0.01 per delivery)");
console.log(pad("design", 34) + lpad("events", 7) + lpad("crashes", 8) + lpad("keys", 6) + lpad("applied", 8) + lpad("lost", 6) + lpad("ledger", 8) + lpad("expected", 9));
for (const design of ["split", "atomic"]) {
  const r = crashTest(design);
  const label = design === "split" ? "insert key, then side effect" : "insert key + side effect, one tx";
  console.log(pad(label, 34) + lpad(r.N, 7) + lpad(r.crashes, 8) + lpad(r.keys, 6) + lpad(r.applied, 8) + lpad(r.lost, 6) + lpad(r.ledger, 8) + lpad(r.expected, 9));
}

console.log("\n== sender retries: 1,000 failed deliveries, receiver back at t=300 s (simulation)");
for (const r of jitter()) console.log(pad(r.mode, 26) + ` attempts ${r.attempts}, peak per second after recovery ${r.peakAfter}, last delivered at t=${r.lastOk} s`);

The unedited output of node webhook-sim.mjs from 2026-10-23 (identical across all three runs):

webhook-sim (simulation, synthetic data)
seed: 20261023
node: v24.20.0
sqlite: 3.53.4
machine: Apple M1 Max darwin arm64
run date: 2026-10-23
events: 10000 (5935 credit, 4065 plan), accounts: 500, same-second plan pairs: 193
chances per delivery: duplicate 0.03, response lost after commit 0.02, concurrent redelivery 0.01, shuffled 0.1 (window 20)

== handlers (simulation)
handler                deliveries   5xx dup events extra applies  lost stale writes stale accts    ledger  expected
a naive                     10633     0        354           368     0           50           5    630300    593500
b check-then-insert         10747   106         65            65     0           23           1    600000    593500
c atomic insert             10633     0          0             0     0           23           1    593500    593500
d order by created>=        10633     0          0             0     0           17           1    593500    593500
d order by created>         10633     0          0             0     0            0          19    593500    593500
d version guard             10633     0          0             0     0            0           0    593500    593500

== traced event evt_000132: add 100 credits to acct_0243
a naive: applied 2x
  first+concurrent attempt 1: 200
  first+concurrent attempt 2: 200
b check-then-insert: applied 2x
  first+concurrent attempt 1: 200
  first+concurrent attempt 2: 500
  retry-after-500 attempt 3: 200
c atomic insert: applied 1x
  first+concurrent attempt 1: 200
  first+concurrent attempt 2: 200

== same-second pairs delivered newer-first: 18 of 193
example acct_0388: evt_000090 sets plan_v1, evt_000091 sets plan_v2, both created 1790000039; arrival order evt_000091 then evt_000090
  a naive: plan_v1 after both arrived
  b check-then-insert: plan_v1 after both arrived
  c atomic insert: plan_v1 after both arrived
  d order by created>=: plan_v1 after both arrived
  d order by created>: plan_v2 after both arrived
  d version guard: plan_v2 after both arrived

== retention: duplicate applies that got past the dedupe store (simulation)
chances: response lost after processing 0.05 per attempt, later attempt unreachable 0.5, manual resend 0.005 (3-day row only, 1 to 15 days later)
Svix published schedule               5 min               268
Svix published schedule               24 h                  4
Svix published schedule               7 days                0
Svix published schedule               30 days               0
Svix published schedule               last retry at  27:35:05
Standard Webhooks example             5 min               247
Standard Webhooks example             24 h                 14
Standard Webhooks example             7 days                0
Standard Webhooks example             30 days               0
Standard Webhooks example             last retry at  75:35:05
3-day window, illustrative intervals  5 min               325
3-day window, illustrative intervals  24 h                 48
3-day window, illustrative intervals  7 days               28
3-day window, illustrative intervals  30 days               0
3-day window, illustrative intervals  last retry at  72:00:00

== crash after dedupe insert, before side effect (simulation, file-backed SQLite, chance 0.01 per delivery)
design                             events crashes  keys applied  lost  ledger expected
insert key, then side effect         2000      15  2000    1985    15  198500   200000
insert key + side effect, one tx     2000      15  2000    2000     0  200000   200000

== sender retries: 1,000 failed deliveries, receiver back at t=300 s (simulation)
exponential, no jitter     attempts 10000, peak per second after recovery 1000, last delivered at t=511 s
exponential, full jitter   attempts 10790, peak per second after recovery 11, last delivered at t=1781 s