Skip to blog content

Experiments & observations

The blog

Things I’m building, learning, and figuring out

50 posts

From the archive

How to evaluate AI agents: task success, tool-call accuracy, cost, and running each task more than once

Agents are non-deterministic and take many steps, so one demo proves nothing. How to build an agent test set, score outcomes in code, check tool calls and trajectories, measure cost and steps, and run each task several times.

Read the post : How to evaluate AI agents: task success, tool-call accuracy, cost, and running each task more than once
From the archive

Single agent or multi-agent? What I learned building a multi-agent financial intelligence system

Multi-agent LLM systems add coordination, cost and new failure modes. When splitting work across agents helps, when one agent with good tools is better, and three design lessons from Cuttlefish: timeouts, failing closed and audit trails.

Read the post : Single agent or multi-agent? What I learned building a multi-agent financial intelligence system
From the archive

What does an AI engineer do? A working definition, and what to look for when hiring one

An AI engineer builds products on top of models: retrieval, agents, evaluation, guardrails and the infrastructure around them. How the role differs from ML engineer and data scientist, the skills that matter, and interview questions for founders hiring one.

Read the post : What does an AI engineer do? A working definition, and what to look for when hiring one
From the archive

My AI-written code detector judged Python with a JavaScript model: lessons from building one for interviews

An interview platform scored candidates' code as AI- or human-written. Reviewing it, I found every language went through the JavaScript model, line-level evidence that was manufactured, and a dataset too small to trust.

Read the post : My AI-written code detector judged Python with a JavaScript model: lessons from building one for interviews
From the archive

Face liveness detection explained: presentation attacks, injection attacks and how they're measured

Liveness detection checks that a live person, not a photo, screen replay or mask, is in front of the camera. Active vs passive liveness, the ISO 30107-3 metrics APCER and BPCER, and why deepfake injection needs a separate defence.

Read the post : Face liveness detection explained: presentation attacks, injection attacks and how they're measured
From the archive

How KYC identity verification works: document capture, OCR, face match and liveness, step by step

What happens between 'take a photo of your ID' and 'verified': document detection, classification, OCR and MRZ checks, face matching, liveness, and the decision. An AI engineer's walkthrough of each stage and where it fails.

Read the post : How KYC identity verification works: document capture, OCR, face match and liveness, step by step
From the archive

How to cut LLM API costs without making the product worse: caching, routing, batching and context

LLM bills are mostly input tokens. The levers that work in production: measure cost per feature, trim context, prompt caching, routing easy requests to cheaper models, batch APIs for offline work, and limits on output and retries.

Read the post : How to cut LLM API costs without making the product worse: caching, routing, batching and context
From the archive

Error budgets and burn rates explained with code: what 14.4 means, and a maintenance-window bug

A 99.9% SLO over 30 days allows 43.2 minutes of failure. How to compute error budget spent and burn rate, why a burn rate of 14.4 over an hour pages someone, and the maintenance-window mismatch I found in my own monitoring console.

Read the post : Error budgets and burn rates explained with code: what 14.4 means, and a maintenance-window bug
From the archive

What is the Model Context Protocol (MCP)? Tools, resources, transports and auth, explained by someone who built a server

MCP is an open protocol that lets AI assistants call tools and read data from other systems. How it works (JSON-RPC, tools, resources, prompts), stdio vs Streamable HTTP, OAuth for remote servers, and what to get right when you build one.

Read the post : What is the Model Context Protocol (MCP)? Tools, resources, transports and auth, explained by someone who built a server