Experiments & observations
The blog
Things I’m building, learning, and figuring out
50 posts
Preparing ID card photos for OCR with OpenCV: find the card, flatten it, and reject blur and glare
Most OCR errors on identity documents come from the photo, not the model. Tested OpenCV code to detect a card's corners, correct perspective to a flat ID-1 rectangle, and score sharpness and glare before OCR runs.
Read the post : Preparing ID card photos for OCR with OpenCV: find the card, flatten it, and reject blur and glareEmbeddings and vector search explained: cosine similarity, HNSW and when to add keyword search
What an embedding is, why cosine similarity measures meaning, how approximate nearest-neighbour indexes such as HNSW make search fast, and why hybrid search with BM25 beats vectors alone for names, codes and exact terms.
Read the post : Embeddings and vector search explained: cosine similarity, HNSW and when to add keyword searchHow to evaluate AI agents: task success, tool-call accuracy, cost, and running each task more than once
Agents are non-deterministic and take many steps, so one demo proves nothing. How to build an agent test set, score outcomes in code, check tool calls and trajectories, measure cost and steps, and run each task several times.
Read the post : How to evaluate AI agents: task success, tool-call accuracy, cost, and running each task more than onceKYC vs AML vs KYB vs CDD: what each means, and where AI fits in each
KYC verifies who a customer is, AML is the wider programme against money laundering, KYB verifies businesses, and CDD/EDD set how deep checks go. A plain-English guide for founders and engineers, with where automation and AI help.
Read the post : KYC vs AML vs KYB vs CDD: what each means, and where AI fits in eachHow LLM tool calling works: the loop, the schema, and the mistakes that break agents
Tool calling (function calling) lets a model ask your code to run a function. The request–execute–return loop step by step, how to write tool schemas and descriptions models use correctly, and the errors, limits and safety checks every agent needs.
Read the post : How LLM tool calling works: the loop, the schema, and the mistakes that break agentsHow NFC passport chip verification works: BAC, PACE, passive and chip authentication
E-passports carry a chip with a signed copy of the data page and photo. How a phone reads it (BAC or PACE using the MRZ), how passive authentication proves the data is genuine, and how chip authentication stops cloning.
Read the post : How NFC passport chip verification works: BAC, PACE, passive and chip authenticationRAG chunking strategies: fixed size, by heading, and what actually changes retrieval quality
How you split documents decides what RAG can retrieve. Fixed-size chunks with overlap, structure-aware chunking by heading (with tested code), keeping titles in each chunk, and how to choose sizes by measuring recall.
Read the post : RAG chunking strategies: fixed size, by heading, and what actually changes retrieval qualitySingle agent or multi-agent? What I learned building a multi-agent financial intelligence system
Multi-agent LLM systems add coordination, cost and new failure modes. When splitting work across agents helps, when one agent with good tools is better, and three design lessons from Cuttlefish: timeouts, failing closed and audit trails.
Read the post : Single agent or multi-agent? What I learned building a multi-agent financial intelligence systemWhat does an AI engineer do? A working definition, and what to look for when hiring one
An AI engineer builds products on top of models: retrieval, agents, evaluation, guardrails and the infrastructure around them. How the role differs from ML engineer and data scientist, the skills that matter, and interview questions for founders hiring one.
Read the post : What does an AI engineer do? A working definition, and what to look for when hiring oneWhat is an LLM gateway? Why teams put LiteLLM between their apps and model providers
An LLM gateway is one API in front of many model providers that adds routing, fallbacks, budgets, keys and cost tracking. What it does, when you need one, and what can go wrong, from a LiteLLM contributor.
Read the post : What is an LLM gateway? Why teams put LiteLLM between their apps and model providersA tamper-evident audit log for AI agents: three bugs in my hash chain, and the fix
Cuttlefish logs every agent action to a SHA-256 hash chain. Reviewing it, I found hashes nobody could recompute, parallel agents forking the chain, and a 'recent entries' query returning the oldest. Here's a version that verifies.
Read the post : A tamper-evident audit log for AI agents: three bugs in my hash chain, and the fixMy AI-written code detector judged Python with a JavaScript model: lessons from building one for interviews
An interview platform scored candidates' code as AI- or human-written. Reviewing it, I found every language went through the JavaScript model, line-level evidence that was manufactured, and a dataset too small to trust.
Read the post : My AI-written code detector judged Python with a JavaScript model: lessons from building one for interviewsFace liveness detection explained: presentation attacks, injection attacks and how they're measured
Liveness detection checks that a live person, not a photo, screen replay or mask, is in front of the camera. Active vs passive liveness, the ISO 30107-3 metrics APCER and BPCER, and why deepfake injection needs a separate defence.
Read the post : Face liveness detection explained: presentation attacks, injection attacks and how they're measuredFace match thresholds: how false accepts trade against false rejects in identity verification
Face verification compares two embeddings and applies a threshold. What FMR and FNMR (FAR and FRR) mean, how to choose a threshold from data with code, and why image quality, ageing and demographics move the numbers.
Read the post : Face match thresholds: how false accepts trade against false rejects in identity verificationHow KYC identity verification works: document capture, OCR, face match and liveness, step by step
What happens between 'take a photo of your ID' and 'verified': document detection, classification, OCR and MRZ checks, face matching, liveness, and the decision. An AI engineer's walkthrough of each stage and where it fails.
Read the post : How KYC identity verification works: document capture, OCR, face match and liveness, step by stepWhy you can't average p95 latencies, and how to combine percentiles correctly
Averaging per-server p95s can halve the latency you report. A worked example with real numbers, how percentile interpolation works on small samples, and the right ways to aggregate: raw samples, histograms or sketches.
Read the post : Why you can't average p95 latencies, and how to combine percentiles correctlyGetting reliable JSON from LLMs: schemas, validation, retries and a deterministic fallback
LLMs return malformed or wrong-shaped JSON more often than you'd like. How to combine provider structured outputs, Pydantic validation, a retry that feeds the error back, business-rule checks and a safe fallback, with tested Python.
Read the post : Getting reliable JSON from LLMs: schemas, validation, retries and a deterministic fallbackReal-time KYC card verification over MQTT: a pub-sub pipeline explained
A three-service KYC pipeline on MQTT: a client publishes ID card data, a verifier checks each field and publishes reasons, an analyst stores and charts results. Design choices and what QoS 1 implies.
Read the post : Real-time KYC card verification over MQTT: a pub-sub pipeline explainedWhy OCR struggles with Arabic, Urdu and other non-Latin scripts, and what helps
Connected letters with up to four shapes, Urdu's sloping Nastaliq baseline, right-to-left text with left-to-right numbers, Eastern Arabic digits, and scripts without spaces. What breaks OCR on identity documents and how pipelines handle it.
Read the post : Why OCR struggles with Arabic, Urdu and other non-Latin scripts, and what helpsHow to build a fully offline RAG chatbot with Ollama, Mistral, ChromaDB and Redis
A document Q&A chatbot for sensitive environments that needs no internet after setup: local embeddings and LLM through Ollama, ChromaDB for vectors, Redis caching, and the settings that matter.
Read the post : How to build a fully offline RAG chatbot with Ollama, Mistral, ChromaDB and RedisHow passport MRZ check digits work, with code to validate them
The two lines of chevrons on a passport are the machine-readable zone. How ICAO 9303 check digits work (weights 7, 3, 1), how to validate the document number, dates and composite digit in Python, and why OCR pipelines rely on them.
Read the post : How passport MRZ check digits work, with code to validate themPrompt injection from scraped data: guarding an LLM that writes outreach emails
LeadForge feeds Google Maps listings into an LLM that drafts cold emails. Anyone can write a listing, so it's untrusted input. The three layers I use: delimited data, output validation with a template fallback, and send gates that fail closed.
Read the post : Prompt injection from scraped data: guarding an LLM that writes outreach emailsHow to evaluate a RAG system: recall@k, MRR and groundedness, with code
Most RAG failures are retrieval failures. How to build a small golden set, measure retrieval with recall@k and MRR, check that answers are grounded in the retrieved text, and use the numbers to tune chunk size and top-k.
Read the post : How to evaluate a RAG system: recall@k, MRR and groundedness, with codeHow to cut LLM API costs without making the product worse: caching, routing, batching and context
LLM bills are mostly input tokens. The levers that work in production: measure cost per feature, trim context, prompt caching, routing easy requests to cheaper models, batch APIs for offline work, and limits on output and retries.
Read the post : How to cut LLM API costs without making the product worse: caching, routing, batching and contextError budgets and burn rates explained with code: what 14.4 means, and a maintenance-window bug
A 99.9% SLO over 30 days allows 43.2 minutes of failure. How to compute error budget spent and burn rate, why a burn rate of 14.4 over an hour pages someone, and the maintenance-window mismatch I found in my own monitoring console.
Read the post : Error budgets and burn rates explained with code: what 14.4 means, and a maintenance-window bugSecuring a voice AI agent's webhook: what my first version got right, and what I'd change
A 24/7 AI phone agent posts every call to a webhook. How HI Agent's backend verifies Vapi, stores repeated call events idempotently, and emails one summary per call, plus the fixes I'd make.
Read the post : Securing a voice AI agent's webhook: what my first version got right, and what I'd changeWhat is the Model Context Protocol (MCP)? Tools, resources, transports and auth, explained by someone who built a server
MCP is an open protocol that lets AI assistants call tools and read data from other systems. How it works (JSON-RPC, tools, resources, prompts), stdio vs Streamable HTTP, OAuth for remote servers, and what to get right when you build one.
Read the post : What is the Model Context Protocol (MCP)? Tools, resources, transports and auth, explained by someone who built a serverYour AI agent thinks its tasks were deleted: how to write tool errors for LLM agents
An agent saw 'task not found' and recreated its schedules, causing duplicates. Tool errors for LLMs must say why and what to do next. Lessons from NanoClaw and an MCP server.
Read the post : Your AI agent thinks its tasks were deleted: how to write tool errors for LLM agentsHow to see every AirPods battery on Linux, and why only one showed: LibrePods
LibrePods brings AirPods features to Linux and Android. With two AirPods connected, the Linux tray showed whichever updated last. A per-device registry fixes it, plus a smarter Android volume duck.
Read the post : How to see every AirPods battery on Linux, and why only one showed: LibrePodsFrom 10 µm to 3 nm: how chips got so small (and why 3 nm isn't 3 nm)
Intel's first microprocessor was built on a 10 µm process. Today's chips are sold as 3 nm. What shrinking transistors buys you, and what the node name really means.
Read the post : From 10 µm to 3 nm: how chips got so small (and why 3 nm isn't 3 nm)How I got my first pull request merged into LiteLLM
What my LiteLLM fix for context compression taught me about making a first open-source contribution to a large project: finding the issue, CI, and getting merged.
Read the post : How I got my first pull request merged into LiteLLMGoogle Workspace vs Zoho Mail for a small startup: is the price gap worth it?
For a 10-person team, Zoho Mail Lite cost about $120–150 a year against roughly $840 for Google Workspace. The trade-offs, and the DNS setup it took.
Read the post : Google Workspace vs Zoho Mail for a small startup: is the price gap worth it?What's inside a GPU? Cores, streaming multiprocessors and the memory problem
A GPU isn't one powerful processor. It's thousands of simple cores, specialised tensor and ray-tracing units, and a memory system built to keep them fed.
Read the post : What's inside a GPU? Cores, streaming multiprocessors and the memory problemHow can deployment platforms afford free tiers?
Servers didn't disappear. Platforms like Vercel and Render stopped renting you machines and started renting execution. Modern Deployments, part 1.
Read the post : How can deployment platforms afford free tiers?Why our AI agent looped forever: a context-compression bug in an LLM gateway
In LiteLLM, content an agent retrieved was compressed back into the same hash, so Claude Code never saw it and kept retrying. How the bug worked and how PR #38591 fixed it.
Read the post : Why our AI agent looped forever: a context-compression bug in an LLM gatewayWhy LLM cost dashboards lie when two deployments share a model name
LiteLLM's auto-router compared only provider and model, so two deployments of the same model at different prices reported $0 savings. Why identity should be the deployment.
Read the post : Why LLM cost dashboards lie when two deployments share a model nameLocalhost is not authentication: fixing a CWE-306 webhook in an AI agent framework
NanoClaw's loopback webhook accepted any local POST, letting other processes forge button clicks (CWE-306, CVSS 6.6). My fix, the maintainers' simpler one, and why theirs was better.
Read the post : Localhost is not authentication: fixing a CWE-306 webhook in an AI agent frameworkStreaming LLM output through a Next.js proxy: why you only get stream_end
Server-Sent Events through a Next.js route handler often arrive all at once or not at all. The headers, runtime flags and client parsing that make real-time LLM streaming work.
Read the post : Streaming LLM output through a Next.js proxy: why you only get stream_endWhy Node.js HTTP backpressure must count bytes, not string length
Node's OutgoingMessage counted buffered strings in UTF-16 code units, so multi-byte bodies were under-counted up to 3× and write() kept saying there was room. My fix, and how I verified it.
Read the post : Why Node.js HTTP backpressure must count bytes, not string lengthHow to build a secure MCP server with OAuth 2.1: lessons from a CRM connector
How I let Claude and other AI clients read a CRM safely: OAuth 2.1 with PKCE, single-use codes, row-level security as the only authorization, and read-only tools.
Read the post : How to build a secure MCP server with OAuth 2.1: lessons from a CRM connectorHow Rosetta, Wine and DXVK run software on machines it was never built for
Rosetta translates CPU instructions, Wine reimplements the Windows API, and DXVK turns DirectX into Vulkan. Three translation layers, three different problems.
Read the post : How Rosetta, Wine and DXVK run software on machines it was never built forWhat is a serverless cold start, really?
A cold start is the time a platform spends creating an execution environment before your handler runs. What happens step by step, and how to make it shorter.
Read the post : What is a serverless cold start, really?Your app doesn't run on a server anymore. It runs on a desired state.
Containers and orchestrators like Kubernetes mean you deploy a workload, not to a machine. What that changes, and why it matters. Modern Deployments, part 2.
Read the post : Your app doesn't run on a server anymore. It runs on a desired state.Labels vs annotations: how one YAML indent made Prometheus scrape Pyroscope twice
A Helm refactor moved prometheus.io/service-monitor from labels to annotations, so the ServiceMonitor stopped excluding Pyroscope's headless service and metrics were duplicated.
Read the post : Labels vs annotations: how one YAML indent made Prometheus scrape Pyroscope twiceHow to stop an LLM from inventing numbers: evidence containment and numeral tracing
In a research-to-LinkedIn pipeline, LLMs extract and write, but code decides what's true: verbatim evidence checks, discarding unsourced numbers, and tracing every numeral back to the paper.
Read the post : How to stop an LLM from inventing numbers: evidence containment and numeral tracingHow does Uber show cars moving in real time? A system design breakdown
Push instead of polling, H3 hexagonal indexing to find nearby drivers, and prediction plus correction to keep cars moving smoothly. How ride-hailing works at scale.
Read the post : How does Uber show cars moving in real time? A system design breakdownWhat actually gets deployed to production? (Not your code)
Production runs an immutable build artifact (a Docker image, bundle or JAR), not your repository. Why that makes rollbacks instant. Modern Deployments, part 4.
Read the post : What actually gets deployed to production? (Not your code)One phone number, two users: normalising WhatsApp sender IDs across Baileys and the Cloud API
Baileys sends 15551234567@s.whatsapp.net, the WhatsApp Cloud API sends 15551234567, so one person became two users with different roles. Canonical IDs and a merge migration fix it.
Read the post : One phone number, two users: normalising WhatsApp sender IDs across Baileys and the Cloud APIWhere do your environment variables go when you deploy?
Secrets pasted into Vercel or Render are not bundled into your build. They take a separate path and are injected at runtime, with one Next.js exception. Part 5.
Read the post : Where do your environment variables go when you deploy?How wide are London's pavements? Measuring 7,908 cross-sections in Whitechapel
No UK dataset records pavement width. I measured left pavement, road and right pavement for 10 Whitechapel streets, then tried four ways to prove myself wrong.
Read the post : How wide are London's pavements? Measuring 7,908 cross-sections in Whitechapel