Production LLM engineering
17 articles, in reading order.
Calling an LLM API in a demo takes one line. Running it for real users means handling rate limits, overloaded providers, malformed output, conversations longer than the context window, streams that break mid-answer, costs that drift, and quality you have to measure rather than assume.
These articles cover each of those problems with tested code. Most of the patterns come from work on LLM gateways, including fixes I contributed to LiteLLM.
Calling LLM APIs reliably
- Retrying LLM API calls: exponential backoff, jitter, Retry-After and model fallbacks
LLM APIs return 429s, 5xx errors and overloaded responses under load. Which errors to retry, full-jitter exponential backoff, honouring Retry-After, a retry budget, and falling back to another model, with tested JavaScript.
- Rate limiting LLM API calls in Node.js: a token bucket for requests and tokens per minute
LLM APIs limit both requests per minute and tokens per minute. A client-side token-bucket limiter in Node.js that waits for both, keeps callers in order, and corrects its token estimate with real usage. Tested code with a fake clock.
- Limiting concurrent async calls in Node.js without a library: a 15-line promise pool
Promise.all over 1,000 items starts 1,000 requests at once, which trips rate limits and exhausts sockets. A tested 15-line mapLimit for Node.js that caps how many run at once, keeps results in input order and doesn't let one failure stop the batch, with notes on choosing the limit for LLM and API calls.
- Limiting concurrency in Python asyncio with a semaphore and per-task timeouts, in the right order
asyncio.gather over 1,000 LLM or API calls starts them all at once. How to cap concurrency with asyncio.Semaphore, give each call its own timeout, keep failures from cancelling the batch, and why putting the timeout outside the semaphore makes tasks time out without ever running. Tested Python 3.11+.
- Getting reliable JSON from LLMs: schemas, validation, retries and a deterministic fallback
LLMs return malformed or wrong-shaped JSON more often than you'd like. How to combine provider structured outputs, Pydantic validation, a retry that feeds the error back, business-rule checks and a safe fallback, with tested Python.
- Trimming chat history to fit the context window without breaking tool calls
Long conversations outgrow the model's context window. Why keeping the last N messages breaks agents by orphaning tool results, how to trim by whole turns within a token budget, and what to do when one turn is too big. Tested Python.
Streaming responses
- Parsing a streamed LLM response in Node.js: Server-Sent Events, split chunks and the CRLF bug
LLM APIs stream tokens as Server-Sent Events, and network chunks don't line up with events. A tested Node.js SSE parser that handles events split mid-JSON, multi-byte characters across chunks, comments, multi-line data and a CRLF split between two chunks.
- Streaming LLM output through a Next.js proxy: why you only get stream_end
Server-Sent Events through a Next.js route handler often arrive all at once or not at all. The headers, runtime flags and client parsing that make real-time LLM streaming work.
- Stop paying for tokens nobody reads: cancel the LLM call when the client disconnects in Node.js
When a user closes the tab mid-answer, a Node.js proxy keeps streaming the LLM response, and you pay for every token. How to abort the upstream request with AbortController when the client goes away, combine it with a timeout using AbortSignal.any, and the test that showed 5 tokens instead of 100.
Gateways, caching and cost
- What is an LLM gateway? Why teams put LiteLLM between their apps and model providers
An LLM gateway is one API in front of many model providers that adds routing, fallbacks, budgets, keys and cost tracking. What it does, when you need one, and what can go wrong, from a LiteLLM contributor.
- How to cut LLM API costs without making the product worse: caching, routing, batching and context
LLM bills are mostly input tokens. The levers that work in production: measure cost per feature, trim context, prompt caching, routing easy requests to cheaper models, batch APIs for offline work, and limits on output and retries.
- Semantic caching for LLM apps: when reusing an answer is safe, and when it's a bug
A semantic cache reuses an LLM answer for a similar question, matched by embedding similarity. How to build one in Python, why the cache key must include the model, system prompt and tenant, how near-misses like 'cancel my order' and 'don't cancel my order' break it, and when not to cache at all.
- Why LLM cost dashboards lie when two deployments share a model name
LiteLLM's auto-router compared only provider and model, so two deployments of the same model at different prices reported $0 savings. Why identity should be the deployment.
- Why our AI agent looped forever: a context-compression bug in an LLM gateway
In LiteLLM, content an agent retrieved was compressed back into the same hash, so Claude Code never saw it and kept retrying. How the bug worked and how PR #38591 fixed it.
- How I got my first pull request merged into LiteLLM
What my LiteLLM fix for context compression taught me about making a first open-source contribution to a large project: finding the issue, CI, and getting merged.
Quality and evaluation
- How to stop an LLM from inventing numbers: evidence containment and numeral tracing
In a research-to-LinkedIn pipeline, LLMs extract and write, but code decides what's true: verbatim evidence checks, discarding unsourced numbers, and tracing every numeral back to the paper.
- Is your LLM judge any good? Measuring agreement with humans using Cohen's kappa
An LLM judge that says 'pass' to everything agrees with humans 90% of the time when 90% of answers pass. Why raw agreement misleads, how Cohen's kappa corrects for chance, how to read the confusion matrix, and a quick test for position bias in pairwise judging. Tested Python.
Common questions
Which LLM API errors should you retry?
Retry 429 rate-limit responses, 5xx server errors, overloaded responses (such as Anthropic's 529) and timeouts, with exponential backoff and full jitter, and honour Retry-After when it's sent. Never retry 400-class errors such as a malformed or over-long request: they fail the same way every time.
How do you stay under an LLM provider's rate limits?
Providers usually limit both requests per minute and tokens per minute. A client-side token bucket for each, reserving an estimated token count before the call and correcting it with the real usage afterwards, avoids most 429s. Cap concurrency separately, and keep retries behind the limiter.
What is an LLM gateway?
A service that sits between your applications and LLM providers, giving them one API while it handles keys, routing between models, retries, fallbacks, rate limits, caching and cost tracking in one place. LiteLLM is a widely used open-source example.
How do you know an LLM judge is reliable?
Grade at least 100 real examples with both humans and the judge, and compare them with Cohen's kappa rather than raw agreement, because a judge that passes everything can agree 90% of the time when 90% of answers pass. For pairwise judges, also swap the answer order to check for position bias.