Building an Observability-Driven Optimization Loop for Coding Agents
From distributed traces to verified performance improvements: an evidence interface, isolated runtime, independent verifier, and three-way experimental protocol.
Engineering notes
Writing
Notes on production AI, distributed systems, ML infrastructure, reliability, and performance.
RSS feedAgent Reliability
Keeping execution authority outside the model: typed proposals, deterministic policy, durable confirmation, revalidation, idempotency and audit.
Read
Stop Asking Coding Agents to Guess What's Wrong
Why Agent-Friendly Codebases need runtime evidence, not just better documentation: traces, profiles, reproducible workloads, and independent verification.
Agentic Without the LLM: Building Deterministic Kubernetes RCA
How I built a bounded autonomous Kubernetes incident investigator where evidence selection and root-cause diagnosis run deterministically — 21/25 exact-root agreement on a frozen blind holdout with zero model calls.
Hard Gates + Frozen Hashes for AI Coding Agents
Making agentic development verifiable rather than merely autonomous: external gates decide whether work is acceptable, and hashed baselines stop the agent from redefining what acceptable means.
A Confirmation Is Not a Boolean: Designing Stateful Agent Workflows
Why "yes" is not always approval, and how interruption, suspension, resume, revalidation, and durable state make consequential agent workflows safe.
The Write May Have Succeeded: Handling Unknown Outcomes in AI Agents
A timeout after a customer-impacting action does not mean the action failed. How action identity, idempotency, database constraints, and reconciliation reduce duplicate writes, and where local guarantees stop.
Memory Is Context, Not Authority
Persistent agent memory is useful only if remembered text cannot become permission. A practical boundary for scoped memory, consent, DLP, TTL, and authority isolation.
Testing AI Agents Without Pretending They Are Deterministic
Deterministic integration tests and real-LLM tests answer different questions. How to use both to verify control-plane invariants and probe semantic quality.
Decision, Authority, Execution: Observability for AI Agents
For consequential agents, intent, authority, and execution must be observable separately. A practical operator view for workflow state, policy, evidence, replay, and traces.
How I Keep Prompt Injection Away from Agent Tools
A defense-in-depth boundary that treats model output, retrieved text, and remembered context as evidence while deterministic software keeps execution authority.
Designing Guardrails for Production AI Agents
A practical execution model for tool-using agents built from typed proposals, deterministic policy, durable confirmation, revalidation, idempotency, and audit.
AI Platform
Model lifecycle as a control-plane problem — progressive delivery, delayed quality feedback, policy-driven promotion and reconciliation.
Designing a Policy-Driven Model Promotion Control Plane
Model deployment is not complete when a container starts. How progressive traffic, delayed ground truth, explicit policy outcomes, rollback, and desired-versus-observed reconciliation make model promotion a control-plane problem.
Governed Text-to-SQL
Turning a natural-language request into a query that is allowed to run: deterministic SQL admission, server-owned grain safety and execution-based evaluation over changing states.
A SUM That Counts the Parent Three Times
A join to a one-to-many child table can silently triple a parent measure, and the query still looks correct. This is why grain safety belongs to the server, not to the model that wrote the SQL.
The Model Wrote the SQL. It Doesn't Get to Run It.
Between a model writing a query and a database running it sits an admission chain that ends in a capability object. The executor runs an accepted QueryPlan, not raw SQL, so passing every gate is the only path to execution.
Your Text-to-SQL Benchmark Is Measuring the Wrong Thing
Exact-match and single-state checks pass wrong SQL that happens to return the right rows. Correctness has to be execution across counterfactual database states, and mutation testing is how you prove the test can tell right from wrong.
Retrieval & RAG
Retrieval as evidence rather than authority: hybrid search, citation integrity, provenance and grounded generation.
RAG Can Provide Evidence. It Cannot Grant Authority.
Retrieved evidence can support an agent answer without becoming an authorization source. How hybrid retrieval, provenance, grounding checks, and bounded abstention keep the knowledge path separate from execution.
63 Rescues, 0 Drops, and 2.4 Seconds
A cross-lingual reranker moved Recall@5 from 0.9563 to 1.0000. The aggregate is the least interesting number in that sentence, and the latency figure means less than it looks like.
Building Citation Integrity into a Production RAG Pipeline
How retrieved source identity, bounded context construction, citation validation, and evaluation turn RAG citations into testable system behavior.
Data Engineering
Shared definitions, lineage and quality gates in the pipelines that feed training, inference and reporting.
A DAG Is a Failure Contract, Not a Schedule
Orchestration tutorials sell the schedule. The parts that decide whether a pipeline survives are retry semantics, write shape, backfill parameterisation and what a half-finished run leaves behind.
One Definition Per Feature
When every pipeline rebuilds the same customer feature, the definitions drift apart silently. Making the feature a shared owned artifact fixes that and creates a change-impact problem that column lineage has to answer.
Distributed Systems
Explicit delivery guarantees, idempotent effects, transactional boundaries and measured service limits.
Building a Reliable Kafka Event Processing Platform
A deep dive into designing, testing, and optimizing an event-driven commerce platform with Kafka, PostgreSQL, Redis, and transactional outbox patterns.
Transactional Outbox: Closing the Database–Kafka Failure Window
How the commerce platform commits business state and a Kafka-ready event together, then publishes that event through a recoverable PostgreSQL outbox.
At-Least-Once Kafka Without Duplicate Side Effects
A concrete Kafka consumer design that combines Redis leases, a PostgreSQL event ledger, transactional writes, bounded retries, and careful offset commits.