From the notebook
Idempotent webhooks in Node.js: handle every event once, even when it's delivered twice
Webhook providers retry, so the same event arrives more than once. How to make a Node.js handler idempotent with a database claim on the event ID, release on failure, take over stale claims, and what 'exactly once' can and can't mean. Tested code with node:sqlite.
Webhook providers deliver at least once: if your endpoint is slow, returns an error, or the response is lost on the way back, they send the same event again. A handler that sends an email or charges a card on every delivery will eventually do it twice. The fix is an idempotent handler: claim the event ID in the database with a unique constraint before doing the work, mark it done afterwards, release the claim if the work fails so the retry can run it, and take over claims that have gone stale because a worker crashed. Below is a tested Node.js implementation using the built-in node:sqlite, and an honest look at what “exactly once” means.
Why do webhooks arrive twice?
Retries are how providers make delivery reliable. Your handler sent the email, but the 200 didn’t reach the provider before its timeout, so from its side the delivery failed. Stripe, GitHub, Twilio and voice platforms such as Vapi all retry on timeouts and non-2xx responses, often for hours. I hit this with a voice agent that emails a call summary: a retried end-of-call-report would send the owner a second email. That case is in securing a voice AI agent’s webhook.
Two things don’t fix it:
- Checking “have I seen this ID?” and then inserting it. Two deliveries can arrive together, both check, both see nothing, both proceed. The check and the claim must be one atomic operation.
- Responding
200before doing the work. Then a crash loses the event entirely, and the provider never retries.
How do you claim an event atomically?
Use the event ID as a primary key, and let the database decide who gets it. The insert either creates the row (you own the event) or conflicts (someone else has it):
import { DatabaseSync } from "node:sqlite"; const db = new DatabaseSync("webhooks.db"); db.exec(`CREATE TABLE IF NOT EXISTS webhook_events ( event_id TEXT PRIMARY KEY, status TEXT NOT NULL CHECK (status IN ('processing', 'done')), updated_at INTEGER NOT NULL )`); const STALE_MS = 5 * 60_000; const claim = db.prepare(`INSERT INTO webhook_events (event_id, status, updated_at) VALUES (?, 'processing', ?) ON CONFLICT (event_id) DO UPDATE SET updated_at = excluded.updated_at WHERE webhook_events.status = 'processing' AND webhook_events.updated_at < ?`); const finish = db.prepare(`UPDATE webhook_events SET status = 'done', updated_at = ? WHERE event_id = ?`); const release = db.prepare(`DELETE FROM webhook_events WHERE event_id = ? AND status = 'processing'`); /** Returns the HTTP status to send. */ export async function handleWebhook(event, sideEffect, now = Date.now()) { const claimed = claim.run(event.id, now, now - STALE_MS).changes === 1; if (!claimed) return 200; // already done, or another worker has it: acknowledge so the sender stops retrying try { await sideEffect(event); finish.run(now, event.id); return 200; } catch (error) { release.run(event.id); // let the sender's retry try again return 500; } }
The ON CONFLICT ... WHERE clause covers the crash case in the same statement: if a row is still processing but hasn’t been touched for five minutes, the worker that claimed it is presumed dead, and the new delivery takes it over. A row that’s done, or freshly processing, is left alone and the delivery is acknowledged. The same SQL works in PostgreSQL, where node:sqlite would be your database client instead.
Does it work?
These checks pass on Node.js 24:
import assert from "node:assert/strict"; // Duplicate delivery: one email, not two. let emails = 0; const sendEmail = async () => void emails++; assert.equal(await handleWebhook({ id: "evt_1" }, sendEmail), 200); assert.equal(await handleWebhook({ id: "evt_1" }, sendEmail), 200); assert.equal(emails, 1); // A failure releases the claim, so the retry runs it. let attempts = 0; const flaky = async () => { if (++attempts === 1) throw new Error("SMTP timeout"); }; assert.equal(await handleWebhook({ id: "evt_2" }, flaky), 500); assert.equal(await handleWebhook({ id: "evt_2" }, flaky), 200); assert.equal(attempts, 2);
The test file also covers the stale case: a claim older than five minutes is taken over, and a fresh one held by another worker isn’t.
Is this exactly once?
Not quite, and it’s worth being precise. The claim makes the side effect run at most once at a time and once after it succeeds. One gap remains: the email is sent, then the process crashes before finish runs. Five minutes later the claim is stale, a retry takes it over, and the email goes out again.
You can’t close that gap in your database alone, because the email provider is a separate system. You close it by making the side effect itself idempotent:
- Pass an idempotency key to the downstream API when it supports one. Stripe accepts an
Idempotency-Keyheader; derive it from the event ID so a repeat is recognised. - Write the result in the same transaction when the side effect is a database change. Then the claim and the change commit or roll back together.
- Make repeats harmless where you can: “set status to shipped” is safe to repeat; “add 1 to the counter” isn’t.
What else should a webhook handler do?
- Verify the signature first, before claiming anything, so forged requests can’t fill your table. Don’t treat “it came from localhost” as proof: see localhost is not authentication.
- Respond quickly. If the work is slow, claim the event, put it on a queue, and return
200; let a worker do the rest with the same claim logic. - Use the provider’s event ID, not a hash of the body. Retries can differ in fields such as a delivery timestamp.
- Clean up old rows after the provider’s retry window has passed.
Summary
- Webhooks are delivered at least once. Plan for duplicates and concurrent deliveries.
- Claim the event ID atomically with a primary key; mark it done after the work; release it on failure.
- Take over claims that have gone stale, so a crashed worker doesn’t block an event forever.
- For true “once”, make the side effect idempotent too, with idempotency keys or a shared transaction.
The same idea applies to message queues: every consumer in a pub-sub KYC pipeline needs it, because brokers redeliver too.