A Confirmation Is Not a Boolean: Designing Stateful Agent Workflows
Why "yes" is not always approval, and how interruption, suspension, resume, revalidation, and durable state make consequential agent workflows safe.
Engineering notes
Writing
Notes on production AI, distributed systems, ML infrastructure, reliability, and performance.
RSS feedAgent Reliability
Keeping execution authority outside the model: typed proposals, deterministic policy, durable confirmation, revalidation, idempotency and audit.
Read
The Write May Have Succeeded: Handling Unknown Outcomes in AI Agents
A timeout after a customer-impacting action does not mean the action failed. How action identity, idempotency, database constraints, and unknown-write handling prevent agents from duplicating business effects.
Memory Is Context, Not Authority
Persistent agent memory is useful only if remembered text cannot become permission. A practical boundary for scoped memory, consent, DLP, TTL, and authority isolation.
Testing AI Agents Without Pretending They Are Deterministic
Deterministic integration tests and real-LLM tests answer different questions. How to use both to verify control-plane invariants and probe semantic quality.
Decision, Authority, Execution: Observability for AI Agents
For consequential agents, intent, authority, and execution must be observable separately. A practical operator view for workflow state, policy, evidence, replay, and traces.
How I Keep Prompt Injection Away from Agent Tools
A defense-in-depth boundary that treats model output, retrieved text, and remembered context as evidence while deterministic software keeps execution authority.
Designing Guardrails for Production AI Agents
A practical execution model for tool-using agents built from typed proposals, deterministic policy, durable confirmation, revalidation, idempotency, and audit.
AI Platform
Model lifecycle as a control-plane problem — progressive delivery, delayed quality feedback, policy-driven promotion and reconciliation.
Designing a Policy-Driven Model Promotion Control Plane
Model deployment is not complete when a container starts. How progressive traffic, delayed ground truth, explicit policy outcomes, rollback, and desired-versus-observed reconciliation make model promotion a control-plane problem.
Retrieval & RAG
Retrieval as evidence rather than authority: hybrid search, citation integrity, provenance and grounded generation.
RAG Can Provide Evidence. It Cannot Grant Authority.
Retrieved evidence can support an agent answer without becoming an authorization source. How hybrid retrieval, provenance, grounding checks, and bounded abstention keep the knowledge path separate from execution.
63 Rescues, 0 Drops, and 2.4 Seconds
A cross-lingual reranker moved Recall@5 from 0.9563 to 1.0000. The aggregate is the least interesting number in that sentence, and the latency figure means less than it looks like.
Building Citation Integrity into a Production RAG Pipeline
How retrieved source identity, bounded context construction, citation validation, and evaluation turn RAG citations into testable system behavior.
Data Engineering
Shared definitions, lineage and quality gates in the pipelines that feed training, inference and reporting.
A DAG Is a Failure Contract, Not a Schedule
Orchestration tutorials sell the schedule. The parts that decide whether a pipeline survives are retry semantics, write shape, backfill parameterisation and what a half-finished run leaves behind.
One Definition Per Feature
When every pipeline rebuilds the same customer feature, the definitions drift apart silently. Making the feature a shared owned artifact fixes that and creates a change-impact problem that column lineage has to answer.
Distributed Systems
Explicit delivery guarantees, idempotent effects, transactional boundaries and measured service limits.
Building a Reliable Kafka Event Processing Platform
A deep dive into designing, testing, and optimizing an event-driven commerce platform with Kafka, PostgreSQL, Redis, and transactional outbox patterns.
Transactional Outbox: Closing the Database–Kafka Failure Window
How the commerce platform commits business state and a Kafka-ready event together, then publishes that event through a recoverable PostgreSQL outbox.
At-Least-Once Kafka Without Duplicate Side Effects
A concrete Kafka consumer design that combines Redis leases, a PostgreSQL event ledger, transactional writes, bounded retries, and careful offset commits.