Blog
Dated notes from the studio: what changed in AI, apps and the web and what to do about it, releases of our open-source tools, and engineering deep dives. Looking for evergreen answers? Those are in the guides.
Deep dives · LLM tokenization explained: BPE, vocabularies and why token counts differHow LLM tokenizers work: BPE, byte-level BPE, WordPiece and Unigram, vocabulary size, and why one text costs different token counts across models.Read the post
Deep dives · Voice agent architecture: speech-to-text, LLM, text-to-speech and the latency budgetHow voice agents work: cascaded STT, LLM and TTS versus speech-to-speech, the latency budget per turn, turn detection, barge-in, telephony and evaluation.Read the post
Deep dives · Building an MCP server: tools, resources, transports and authorizationBuild an MCP server on the 2026-07-28 spec: tools, resources, prompts, stdio vs Streamable HTTP, stateless requests, OAuth 2.1 and security rules.Read the post
Deep dives · Sandboxing AI-generated code: containers, gVisor, microVMs and WebAssemblyHow to sandbox code an AI agent writes and runs: the threat model, containers, seccomp, gVisor, Firecracker microVMs, Kata, WebAssembly and practical limits.Read the post
Deep dives · System design: file uploads with presigned URLs, multipart uploads and processingHow to design file uploads: presigned URLs straight to object storage, multipart and resumable uploads, magic-number checks, malware scans and a CDN.Read the post
Deep dives · Zero-downtime database migrations: expand and contract, backfills and locksHow to change a PostgreSQL schema without downtime: expand and contract, safe recipes for columns and indexes, batched backfills, lock_timeout and rollback.Read the post
Deep dives · LoRA and fine-tuning explained: when to fine-tune an LLM and how adapters workWhen fine-tuning an LLM beats prompting or RAG, and how LoRA and QLoRA work: low-rank updates, rank and alpha, 4-bit training, adapter serving and data.Read the post
Deep dives · LLM guardrails architecture: input checks, output checks and policy layersHow LLM guardrails fit together: input checks, output checks, tool-call policy, rules vs classifiers vs LLM judges, streaming checks and fail-closed design.Read the post
Deep dives · Text-to-SQL agents: how LLMs answer questions from your database, and how to keep them safeHow text-to-SQL agents turn a question into SQL: schema retrieval, generation, validation and self-correction, and why the database must enforce the limits.Read the post
Deep dives · System design: search autocomplete (typeahead)How to design search autocomplete: a trie with top-k per node, prefix indexes, suggestions from query logs, edge caching, typo tolerance and the client.Read the post
Deep dives · Distributed locks: leases, fencing tokens and when not to use oneHow distributed locks fail under pauses, clock drift and slow networks, why leases need fencing tokens, the Redlock debate, and the alternatives to a lock.Read the post
Deep dives · Feature flags: how flag systems work, and how to keep them from becoming debtHow feature flag systems evaluate rules, bucket users for percentage rollouts, stream updates, fail safely and stay auditable, and how to clean up stale flags.Read the post
Deep dives · AI agent memory: short-term context, long-term stores and what to forgetHow AI agent memory works: context windows, rolling summaries, retrieval by relevance, recency and importance, write policies, MemGPT, privacy and evaluation.Read the post
Deep dives · Multi-agent orchestration patterns: orchestrator-worker, pipelines, handoffs and debateMulti-agent orchestration patterns compared: pipelines, routing, orchestrator-workers, voting, evaluator loops, handoffs and debate, plus costs and failures.Read the post
Deep dives · How LLM inference serving works: prefill, decode, continuous batching and throughputHow LLM serving works: prefill and decode, TTFT and inter-token latency, continuous batching, chunked prefill, disaggregation and speculative decoding.Read the post
Deep dives · Event sourcing and CQRS: storing what happened, not just the current stateHow event sourcing and CQRS work: event streams and replay, projections, snapshots, upcasting, idempotent read models, GDPR deletion and when to avoid them.Read the post
Deep dives · Consistency models explained: CAP, PACELC, linearizability and eventual consistencyWhat the CAP theorem really proves, PACELC, linearizable, causal and eventual consistency, quorums where R + W > N, and how to pick a model for each feature.Read the post
Deep dives · System design: a notification system for email, push and SMSHow to design notifications: preferences, templates, quiet hours by time zone, dedupe keys, per-channel queues, retries, push tokens, bounces and unsubscribe.Read the post
Releases · site-agent: let visitors ask your website to do things, with your rulesWe published site-agent, an open-source tag that lets visitors ask your site to do things and calls only the WebMCP tools your page declared.Read the post
News · Have an AI feature on your site? What the EU AI Act asks of it now, and what’s due by 2 December 2026Article 50 of the EU AI Act has applied since 2 August 2026. Older generative AI systems have until 2 December 2026 to mark their output. A checklist.Read the post
Releases · Run our site, security and AI-config checks in GitHub Actionssite-audit, port-audit and agent-config-audit now ship as GitHub Actions on the Marketplace: one step scans, reports in the job summary and fails on Fix now.Read the post
Deep dives · RAG explained: how retrieval-augmented generation works, step by stepHow retrieval-augmented generation works, step by step: parsing, chunking, embedding, indexing, retrieval, reranking and prompting, with a Python sketch.Read the post
Deep dives · Chunking strategies for RAG: fixed, recursive, structure-aware, semantic and moreFixed-size, recursive, structure-aware, semantic, parent-child, contextual and late chunking for RAG: how each splits text, its trade-offs, and how to test it.Read the post
Deep dives · Hybrid search: combining BM25 and vector search, and how rank fusion worksHow BM25 and vector search fail in different ways, how reciprocal rank fusion and weighted score fusion merge them, and how to tune and test the result.Read the post
Deep dives · How to evaluate a RAG system: retrieval metrics, answer faithfulness and a test setEvaluate RAG in two halves: retrieval with recall@k, MRR and nDCG, and answers with faithfulness and relevance, on a labeled test set that runs in CI.Read the post
Deep dives · How LLM tool calling works: the agent loop, step by stepHow LLM tool calling works: tool definitions, what the model returns, the agent loop, parallel calls, tool_choice, strict schemas, errors, safety and MCP.Read the post
Deep dives · System design: an AI chat assistant for a websiteA system design for a website AI assistant: ingestion on publish, hybrid retrieval, SSE streaming, caching layers, guardrails, evaluation and failure modes.Read the post
Deep dives · Designing a rate limiter: token bucket, sliding window and limits across many serversHow fixed window, sliding window and token bucket rate limiters work, how to hold one limit across many servers, and how to answer with 429 and Retry-After.Read the post
Deep dives · Idempotency and job queues: at-least-once delivery, retries and exactly-once effectsWhy queues deliver at least once, and how idempotency keys, dedupe tables, the outbox pattern and dead-letter queues give you exactly-once effects.Read the post
Deep dives · Caching strategies: cache-aside, write-through, write-back, TTLs and invalidationCache-aside, read-through, write-through and write-back compared, with TTL jitter, invalidation, stampede protection, eviction and HTTP caching headers.Read the post
Deep dives · Database indexing explained: B-trees, composite and covering indexes, and reading a query planHow B-tree indexes work, what they cost, how to order composite indexes, when covering and partial indexes help, and how to read a PostgreSQL query plan.Read the post
Deep dives · Text embeddings explained: vectors, similarity, dimensions and quantizationHow text embeddings work: contrastive training, cosine vs dot product, Matryoshka dimensions, token limits, int8 and binary quantization, and model choice.Read the post
Deep dives · Prompt caching and the KV cache: how reusing a prompt prefix cuts latency and costHow the KV cache and prompt caching work: prefill and decode, prefix reuse in vLLM and SGLang, OpenAI, Anthropic and Gemini caching, and how to get hits.Read the post
Deep dives · Vector search indexes explained: HNSW, IVF and product quantizationHow approximate nearest neighbor indexes work: HNSW graphs, IVF clustering, product quantization, DiskANN, filtered search, and how to benchmark recall.Read the post
Deep dives · Reranking for search and RAG: cross-encoders, ColBERT and LLM rerankersHow rerankers work in search and RAG: bi-encoders vs cross-encoders, ColBERT late interaction, LLM rerankers, candidate depth, score thresholds and evaluation.Read the post
Deep dives · How structured outputs work: constrained decoding, grammars and JSON SchemaHow LLM structured outputs work: logit masking, JSON Schema compiled to automata, token boundaries, provider strict-mode limits, quality and retries.Read the post
Deep dives · LLM observability: tracing, metrics and evals for AI features in productionWhat to record on every LLM call, how to trace agent runs with the OpenTelemetry GenAI conventions, redaction, sampling, alerts and online evals.Read the post
Deep dives · System design: a reliable webhook delivery systemHow to design webhook delivery: an outbox and a queue, retries with backoff and jitter, HMAC signatures, SSRF checks, dead letters, replay and delivery logs.Read the post
Deep dives · Sharding and consistent hashing: how data is split across many serversWhen to shard, how to pick a shard key, and how consistent hashing, virtual nodes, rendezvous and jump hash keep keys in place when servers are added.Read the post