Commerce events
- Registrations, browsing, carts, orders and payments
- Refund journeys with partition-scoped causal ordering
- Versioned event envelopes with one validation boundary
Distributed Systems / Streaming
Real-Time Commerce Platform
Production-oriented event-driven commerce platform for asynchronous customer journeys. Kafka may deliver an event more than once; the processor makes durable business effects safe to replay with layered idempotency, transactional persistence and bounded failure handling.
View repository Production-oriented reference implementation · local evidence, not a production SLA
- 3 workers · 3 partitions~1,050 evt/s
- Sustainable isolated capacityHighest clearly sustainable rate in the local Kafka → processor → persistence benchmark; ~1,075 evt/s degraded repeatably.
- 125,669 → 4,385 calls28.6×
- Fewer offset commitsBounded contiguous per-partition batching moved the earlier sustainable boundary from ~750 to ~900 evt/s.
- 10.897 ms before indexing0.253 ms
- Recent-payment lookupMeasured PostgreSQL execution time after query-plan-driven indexing, not end-to-end latency.
Delivery can repeat. Durable business effects must not.
The platform models registrations, browsing, carts, orders, payments and refunds as versioned events. The important guarantee is not that Kafka never redelivers. It is that a replay cannot create a second order effect, fraud evaluation or derived alert.
PostgreSQL owns durable business truth. Redis coordinates active processing but is deliberately reconstructible. Kafka offsets advance only after terminal handling, so delivery semantics and business-effect semantics stay explicit instead of being collapsed into an "exactly once" slogan.
What the platform does
This is a runnable commerce reference system with an interactive Demo Control Center, deterministic fraud evaluation and an inspectable event processor path.
Durable outcomes
- PostgreSQL business and fraud state
- Deterministic fraud evaluations and explainable alerts
- Transactional outbox for derived Kafka events
Failure and operations
- Bounded retry and confirmed DLQ handling
- Interactive scenarios, run history and health views
- Prometheus metrics and provisioned Grafana dashboards
The write may have succeeded
Consider the uncomfortable but normal timing window: the PostgreSQL transaction commits, then the processor crashes before Redis completion and the Kafka offset commit. The caller sees an interruption, but the business effect already exists.
- 01
Event delivered
Kafka delivers a commerce event to the processor with manual offset management and partition-scoped ordering.
- 02
Lease and transaction
Redis coordinates active ownership while PostgreSQL records the event identity, business effect, fraud result and outbox row in one transaction.
- 03
Crash after commit
The process stops before Redis completion and the Kafka offset commit. The database effect is already durable, but the offset remains uncommitted.
- 04
Redelivery is expected
The lease expires and Kafka redelivers the event. The durable processed_events ledger recognizes the same event and its digest.
- 05
Safe replay
Business writes, fraud evaluation and outbox insertion do not run a second time. The duplicate is acknowledged without creating a second effect.
- 06
Derived event recovery
A separate outbox publisher retries committed fraud alerts. A pending row remains available when publisher delivery is interrupted.
An exception tells the caller what it observed, not necessarily what PostgreSQL committed. That is why writes use stable event identity, durable uniqueness and transaction boundaries rather than blind retries.
At-least-once delivery vs durable effects
| Kafka delivery semantics | Application correctness |
|---|---|
|
|
The offset is a delivery cursor, not proof that the business transaction happened exactly once. The durable ledger and the database transaction provide the replay boundary.
Architecture
Browser-driven scenarios use the same Kafka and processor path as generated commerce events. Redis coordinates active ownership; PostgreSQL commits business, fraud and outbox state; a separate publisher emits committed fraud alerts; the DLQ receives invalid or retry-exhausted records.
Scenario to durable processing
The interactive demo reaches the same Kafka and processor path used by generated commerce events.
Browser controls Demo Control Web. Demo Control Web then connects to Demo Control API. Demo Control API starts scenario Scenario Runner. Scenario Runner publishes commerce.events. commerce.events at least once Event Processor.
- ClientBrowserbounded scenarios
- ClientDemo Control WebNext.js
- ServiceDemo Control APIFastAPI
- ServiceScenario Runnershared event generator
- Messagingcommerce.eventspartitioned Kafka topic
- ServiceEvent Processorpartition-scoped handling
Processor state boundaries
Operational coordination stays reconstructible; durable identity and business effects share one database boundary.
Event Processor coordinates Redis. Event Processor commits durable effects PostgreSQL.
- ServiceEvent Processorterminal handling
- PersistenceRedisleases · completion markersPersistencePostgreSQLdurable system of record
- business + fraud state
- processed identity + outbox
Confirmed failure path
The processor owns dead-letter publication after validation failure or bounded retry exhaustion.
Event Processor publishes invalid / exhausted commerce.events.dlq.
- ServiceEvent Processorclassifies terminal failure
- Messagingcommerce.events.dlqconfirmed DLQ delivery
Derived fraud alerts
Committed outbox rows publish independently from the source-event transaction.
PostgreSQL outbox claims Fraud Outbox Publisher. Fraud Outbox Publisher publishes commerce.fraud-alerts.
- PersistencePostgreSQL outboxcommitted with source effects
- ServiceFraud Outbox Publisherindependent delivery loop
- Messagingcommerce.fraud-alertsKafka topic
Non-critical observability
Application metrics and infrastructure exporters are scraped without changing processing outcomes.
Services + exporters scraped by Prometheus. Prometheus visualized in Grafana.
- ObservabilityServices + exportersbounded-label metrics
- ObservabilityPrometheusplatform signals
- ObservabilityGrafanaprovisioned dashboards
Architecture notes
- The Event Processor—not the main Kafka topic—publishes invalid or exhausted records to commerce.events.dlq.
- PostgreSQL is the durable system of record; Redis contains reconstructible coordination state.
- Accepted event effects, fraud decisions, processed identity and outbox rows commit in one PostgreSQL transaction.
- Kafka offsets commit in bounded per-partition batches (50 records or 100ms, whichever comes first), with a synchronous flush on idle, rebalance and shutdown; delivery is at least once and ordering is partition scoped.
- Two independently evidenced changes — bounded offset-commit batching, then query-plan-driven PostgreSQL indexing — moved the isolated three-worker pipeline's sustainable capacity from ~750 to ~1,050 events/s (~40%) without weakening at-least-once correctness.
Four reliability patterns
Idempotency is layered on purpose
Redis is the fast coordination path with token-checked leases. PostgreSQL's processed_events ledger is the durable safety net when a lease expires, Redis loses state, or a crash occurs after commit. Neither layer is treated as sufficient by itself.
The transaction owns the business boundary
The ledger insert, commerce repositories, fraud evaluation, alert and transactional-outbox row share one PostgreSQL transaction. The alert cannot commit without the outbox fact that it needs to be published.
Failure handling is bounded and explicit
Only classified transient failures retry with capped backoff. Invalid or exhausted records follow a confirmed DLQ path, while contiguous terminal offsets prevent the consumer from committing past an unresolved gap.
Performance follows evidence, not a headline
The project separates Demo full-path throughput from the isolated processor benchmark. Offset-commit batching moved the isolated boundary from about 750 to 900 events/s; query-plan-aware indexes and a fresh sweep established about 1,050 events/s as the highest clearly sustainable point.
Repository proof
The project keeps the mechanics visible in repository diagrams. These are the actual architecture artifacts, linked to their original GitHub files so the full-size detail remains available on smaller screens.
Measured evidence
The numbers below come from different paths and experiments. They are kept separate so a local isolated capacity result is not mistaken for Demo Control throughput or a production SLA.
| Evidence slice | Current result | What it means |
|---|---|---|
| Isolated sustainable capacity | ~750 → ~1,050 evt/s (+40%) | Three workers and three Kafka partitions on the isolated Kafka → processor → persistence path. ~1,050 evt/s stayed bounded and correct across all retained repeats; ~1,075 evt/s was the first repeatably degraded rate. |
| Offset-commit optimization | 125,669 → 4,385 commit calls | Bounded per-partition contiguous batching reduced commit calls by about 28.6x and moved the earlier sustainable boundary from ~750 to ~900 evt/s. |
| Query-plan evidence | 10.897 → 0.253 ms | Measured recent-payment lookup after aligning PostgreSQL indexes with the equality, range and descending-timestamp access pattern. This is query execution time, not end-to-end latency. |
| Demo full path | 49.843 → 97.934 evt/s median | The interactive Demo Control path improved after generator hot-path and pacing fixes. It is intentionally separate from the isolated processor capacity result. |
Observability for the actual system
The Demo Control Center exposes run progress, outcomes, infrastructure health, fraud decisions, retries and DLQ state. Prometheus and Grafana add platform-wide views for processed rates, latency, consumer lag, persistence outcomes, outbox state and exporter health.
Run outcomeDurable stateLag / latencyFraud / outbox / DLQ
Metrics use bounded labels. Customer identifiers and run IDs are kept out of Prometheus labels; run-specific counts come from PostgreSQL-backed projections.
Current implementation
BackendPython 3.12 + FastAPI + Pydantic v2 + psycopg 3
MessagingApache Kafka 3.9 KRaft + confluent-kafka
PersistencePostgreSQL 17 + Redis 7
FrontendNext.js App Router + React + Tailwind + Recharts
ObservabilityPrometheus + Grafana + exporters + prometheus-client
VerificationRuff + mypy + pytest + Vitest
RuntimeDocker Compose + GNU Make
Reference implementation with explicit limits
This is a production-oriented reference implementation for inspecting streaming failure modes. It does not claim production availability, exactly-once delivery, cloud deployment, general Kafka/PostgreSQL capacity or a production fraud decision service.
The measured results come from a local Docker environment with one Kafka broker and the documented workload. The disruptive outage test was not run, and the isolated benchmark must not be read as the throughput of the interactive Demo Control path.