Reliable backends in Node.js and Python

12 articles, in reading order.

Most production incidents aren't exotic. A webhook arrives twice, a retry charges a card again, an event is lost between a database commit and a queue, a deploy drops requests, or a dashboard averages percentiles. Each has a well-known fix.

These articles implement those fixes in Node.js and Python, each with tests that reproduce the failure first.

Doing things exactly once

  1. Idempotent webhooks in Node.js: handle every event once, even when it's delivered twice

    Webhook providers retry, so the same event arrives more than once. How to make a Node.js handler idempotent with a database claim on the event ID, release on failure, take over stale claims, and what 'exactly once' can and can't mean. Tested code with node:sqlite.

  2. Idempotency keys for POST APIs in Node.js: retry a payment without charging twice

    A client retries a POST after a timeout, and the customer is charged twice. How Idempotency-Key headers work: store the first response and replay it, reject the same key with a different body, return 409 while the first request runs, and release the key on server errors. Tested Node.js with node:sqlite.

  3. The transactional outbox pattern in Node.js: never lose an event when the database commits

    Saving a change and publishing an event are two systems, and one can fail after the other succeeds. How the transactional outbox writes both in one database transaction, then relays events in order with retries. Tested Node.js with node:sqlite, including the ordering bug my first relay had.

Running servers

  1. Graceful shutdown in Node.js: draining HTTP requests on SIGTERM without dropping traffic

    When Kubernetes or a deploy stops a Node.js server, server.close() alone can hang on keep-alive sockets or cut requests off mid-response. The order that works: fail readiness, wait, stop accepting, close idle connections, let in-flight requests finish, then force a deadline. Tested with node:http.

  2. Why Node.js HTTP backpressure must count bytes, not string length

    Node's OutgoingMessage counted buffered strings in UTF-16 code units, so multi-byte bodies were under-counted up to 3× and write() kept saying there was room. My fix, and how I verified it.

  3. Request IDs in Node.js logs with AsyncLocalStorage: trace one request through every await

    Concurrent requests interleave their log lines. How AsyncLocalStorage attaches a request ID to every log line without passing it through each function, forwards it to downstream services, and rejects unsafe incoming IDs. Tested with 20 concurrent requests on Node.js.

  4. Verifying JWTs in Node.js: alg none, algorithm confusion, audience checks and clock skew

    The JWT checks that get skipped and the attacks they allow: trusting the token's alg header, comparing signatures with ===, forgetting the audience, and missing exp. A small tested HS256 verifier with node:crypto that shows each check, and why production code should still use a maintained library.

Time and measurement

  1. Scheduling a job at 9am local time in Python: DST, skipped hours and repeated hours

    A job scheduled as '9am, then every 24 hours in UTC' runs at 10am after daylight saving starts. How to compute the next local run with zoneinfo, what happens to 02:30 when clocks jump forward and to 01:30 when they fall back, and why Pakistan's fixed offset hides the bug. Tested Python.

  2. p50 vs p90 vs p99 latency explained: what each percentile tells you, and why the tail matters more than it looks

    p50 is the typical request, p90 and p99 are the slow ones. A tested example where the mean describes nobody, p99 is 20 times p50, and a page that makes 20 backend calls is as slow as the per-call p90 half the time.

  3. Why you can't average p95 latencies, and how to combine percentiles correctly

    Averaging per-server p95s can halve the latency you report. A worked example with real numbers, how percentile interpolation works on small samples, and the right ways to aggregate: raw samples, histograms or sketches.

  4. Error budgets and burn rates explained with code: what 14.4 means, and a maintenance-window bug

    A 99.9% SLO over 30 days allows 43.2 minutes of failure. How to compute error budget spent and burn rate, why a burn rate of 14.4 over an hour pages someone, and the maintenance-window mismatch I found in my own monitoring console.

  5. Labels vs annotations: how one YAML indent made Prometheus scrape Pyroscope twice

    A Helm refactor moved prometheus.io/service-monitor from labels to annotations, so the ServiceMonitor stopped excluding Pyroscope's headless service and metrics were duplicated.

Common questions

How do you make a webhook handler idempotent?

Claim the event ID in the database with a unique constraint before doing the work, mark it done afterwards, release the claim if the work fails so a retry can run, and take over claims that have gone stale because a worker crashed.

What is the transactional outbox pattern?

Write a state change and the event announcing it into the same database transaction, using an outbox table, so both are saved or neither is. A relay then publishes events from the table in order, retrying until they succeed. Delivery is at least once, so consumers must be idempotent.

Why can't you average p95 latencies?

Percentiles don't combine by averaging: the p95 of two servers together depends on their full distributions and how many requests each handled. Merge the raw samples, histograms with shared buckets, or a mergeable sketch such as t-digest, and compute the percentile from the combined data.

Other guides