Progressive delivery
- Weighted stable/canary routing
- 10% → 25% → 50% → 100% traffic stages
- Promotion and rollback through explicit state transitions
MLOps / AI Platform
ModelOps Control Plane
Policy-driven ML release control plane for progressive canary traffic, delayed ground-truth quality gates, automated promotion and rollback, and continuous desired-versus-observed router reconciliation.
View repository Production-oriented control-plane reference implementation · local Compose scope
- Progressive delivery10 → 25 → 50 → 100%
- Automated canary progressionThe real stateless worker advances a genuinely healthy candidate through every traffic stage and promotes it on a live minimum_recall PASS.
- Verification279 TESTS
- Backend regression suitepytest alongside Ruff and mypy --strict, ~91% statement coverage, run on every push.
- System evidence6 REAL-STACK SCENARIOS
- Integration CICI boots the real nine-container stack and exercises worker-driven promotion, rollback and router-restart recovery.
- Control loopDESIRED ↔ OBSERVED
- Router reconciliationThe database's durable desired routing state is continuously reconciled against the router's restart-losable observed state.
Model promotion is a control loop, not a deploy button.
A model can be healthy as a process and still be wrong for production traffic. This platform exposes a candidate gradually, waits for the evidence that can actually judge it, and records the decision that follows.
Reliability signals and model-quality signals are deliberately different. Delayed labels are allowed to mature, insufficient data stays INCONCLUSIVE, and the worker acts through the same control-plane API that an operator uses.
What the platform does
ModelOps Control Plane turns a model release into an inspectable, policy-driven rollout with real traffic, real label ingestion and an explicit recovery path.
Evidence-aware policy
- Latency and error-rate reliability checks
- Delayed ground-truth label ingestion by prediction_id
- Matured quality windows with coverage and positive-label gates
Control loop and operations
- Stateless worker using the same API as an operator
- Desired-versus-observed router reconciliation
- Explainable deployment timeline and manual automation hold
One rollout, judged when the evidence is ready
The quality-failure scenario makes the control loop concrete. A deliberately weak canary receives real routed traffic, but the worker does not call its result decisive until delayed ground truth is sufficient.
- 01
Candidate at 10%
A new model version enters a weighted canary rollout while the stable version keeps the remaining traffic.
- 02
Two windows, two questions
Fresh traffic measures reliability; an older matured window gives delayed ground truth time to arrive before quality is judged.
- 03
Not enough evidence yet
Missing label coverage or too few positive labels produces INCONCLUSIVE. The worker does not mistake a thin sample for a healthy model.
- 04
Quality signal becomes decisive
Once sufficiency gates pass, minimum_recall can genuinely resolve to PASS or FAIL using labels joined to the exact prediction_id.
- 05
Worker acts through the control plane
PASS advances 10% → 25% → 50% → 100%; FAIL triggers rollback; INCONCLUSIVE freezes for human review.
- 06
Timeline records why
Policy snapshots, state transitions and worker actions form one chronological audit story instead of a final status without context.
The important distinction is between “not enough evidence yet” and “this candidate failed.” A policy engine that collapses them will either promote too early or roll back a healthy model on a thin sample.
Control plane vs serving plane
The control plane decides the desired traffic state. The router owns its static version-to-host mapping and reports what it actually observes. Model serving runs predictions; it does not decide whether a version deserves more traffic.
| Control plane owns | Serving / router owns |
|---|---|
|
|
Desired state is durable intent. Observed router state is a cache that can drift. The reconciliation loop exists because a successful decision and a successful push are not the same event.
Architecture
Client traffic enters a weighted router, while the control plane collects metrics and delayed labels. A stateless worker evaluates fresh reliability and matured quality windows, then advances or resolves the rollout through the same API as a human. Reconciliation closes the gap between database intent and router reality.
Prediction / traffic path
The router owns the version → host mapping; an unhealthy selected target returns an error rather than silently failing over.
Client Traffic requests Weighted Router. Weighted Router routes Stable Model (current version). Weighted Router routes Canary Model (candidate version, no fallback). Stable Model and Canary Model emits Control Plane.
- ClientClient Trafficprediction request
- ServiceWeighted Routerowns version → host mapping ({version, weight} only, never host/port)
- ServiceStable Modelcurrent versionServiceCanary Modelcandidate version, no fallback
- ControlControl Planeprediction_id-tagged metric, via POST /metrics (fire-and-forget)
Delayed ground-truth / quality path
Label and metric writes are independent; a GroundTruthLabel is durable even before its matching PredictionMetric arrives.
Delayed Label Source ingests GroundTruthLabel. GroundTruthLabel join by prediction_id Quality Aggregation. Quality Aggregation summarizes Quality Summary.
- BoundaryDelayed Label Sourcesynthetic ground truth, via POST /api/labels(/batch)
- PersistenceGroundTruthLabelwritten unconditionally, durable
- AnalyzerQuality Aggregationjoined against PredictionMetric at read time
- OutputQuality Summaryrecall over matured window
Policy / automated rollout loop
A separate stateless worker acts only through the same Control Plane endpoints available to an operator.
Automation Worker evaluates Two Evaluation Windows. Two Evaluation Windows feeds Policy Engine. Policy Engine verdict PASS (advance · 10% → 25% → 50% → 100%). Policy Engine verdict FAIL (automatic rollback). Policy Engine verdict INCONCLUSIVE (freeze for manual review).
- ServiceAutomation Workerrestart-safe polling loop
- AnalyzerTwo Evaluation Windowsfresh reliability · matured quality
- AnalyzerPolicy Engine7 checks · FAIL beats INCONCLUSIVE beats PASS
- ControlPASSadvance · 10% → 25% → 50% → 100%ControlFAILautomatic rollbackControlINCONCLUSIVEfreeze for manual review
Desired / observed reconciliation
The database's desired state commits first; the router push is best-effort and repaired on drift, not assumed to always land.
Deployment + TrafficAllocation pushes Best-Effort Router Push. Best-Effort Router Push updates Router Observed Config. Router Observed Config reconciles Reconcile Tick.
- PersistenceDeployment + TrafficAllocationdurable desired state, model-scoped generation — commits before router push
- ControlBest-Effort Router PushPUT /router/config
- ServiceRouter Observed Configin-memory, restart-losable, rejects stale generation
- ControlReconcile Tickworker-triggered, repairs drift
Operator / audit
Manual actions use the same endpoints the worker does; every action and policy verdict lands on one merged timeline.
Dashboard calls Control Plane API. Control Plane API records Deployment Timeline.
- ClientDashboardpause / resume / promote / rollback
- ControlControl Plane APImanual or automated actor, same endpoints as the worker
- OutputDeployment Timelineevents + policy snapshots merged
Architecture notes
- The database is the durable desired routing state; the router's config is an in-memory, restart-losable cache the control plane pushes best-effort after committing that decision.
- TrafficAllocation revisions are scoped per model (RoutingGeneration), not per deployment, so the router rejects an equal-or-stale push even when it comes from an already-superseded rollout.
- SQLAlchemy optimistic concurrency (a version column bumped on every commit) and a DB-level partial unique index together stop a losing concurrent action from corrupting a rollout and cap each model at one unresolved deployment at a time — INCONCLUSIVE counts as unresolved too.
- INCONCLUSIVE means frozen for manual review, not a silent revert to the previous traffic split — its allocation stays the router's authoritative desired state until a human resolves it, the same as a completed PROMOTED or ROLLED_BACK rollout.
- The single-router, SQLite, local-compose scope is deliberate, not an oversight — Kubernetes, PostgreSQL, Kafka-based metrics, MLflow and auth are documented production-evolution steps, not implemented here.
Four reliability patterns
Delayed labels are evidence, not a backfill detail
Predictions mint a stable prediction_id and labels arrive through a separate idempotent API. GroundTruthLabel is written even when its metric has not arrived yet, then both are joined at read time. Quality gates wait for label coverage and minimum positive labels before trusting recall.
PASS, FAIL and INCONCLUSIVE are different states
Reliability and quality checks produce explicit persisted evaluations. FAIL beats INCONCLUSIVE, which beats PASS; insufficient evidence cannot silently become approval, and a frozen rollout remains visible for human resolution.
Desired state converges to observed state
The database commits the desired allocation first. The router receives a best-effort push and keeps only observed in-memory state. A worker-triggered reconcile tick compares revisions and repairs drift after a router restart or transient push failure.
Concurrency and stale writes are rejected
Optimistic concurrency, a partial unique index for one unresolved deployment per model, and model-scoped routing generations prevent concurrent actions or delayed pushes from corrupting the rollout currently in charge.
Product proof from the current dashboard
These are curated repository screenshots from the public `origin/main` walkthrough. Each image links back to the source file so the full-size timeline remains available on desktop and mobile.
Selected release evidence
The evidence below describes different parts of the control loop: automated rollout, quality rollback, restart recovery and deterministic verification. They are not merged into one synthetic score.
| Evidence slice | Current result | What it means |
|---|---|---|
| Automated healthy rollout | 10% → 25% → 50% → 100% | A real-stack CI scenario waits for the stateless worker to advance a healthy canary on live routed traffic and delayed labels, then promote it on a genuine minimum_recall PASS. |
| Quality-driven rollback | Recall FAIL → automatic rollback | A deliberately weak canary follows the same delayed label path; after sufficiency gates clear, minimum_recall fails and the worker rolls the deployment back. |
| Restart-safe routing | CI scenarios 5–6 | The router is restarted during a rollout and after a terminal promotion. Reconciliation or startup sync restores the desired allocation without a human replay. |
| Backend verification | 279 tests · ~91% coverage | Ruff, mypy --strict and pytest run alongside a separate integration job that boots the real nine-container stack and exercises six scenarios. |
Operator observability
The dashboard does not stop at “promoted” or “rolled back.” It shows traffic allocation, latency and error-rate comparisons, label coverage, positive-label counts, policy explanations, desired and observed router revisions, and one merged deployment timeline.
TrafficLabelsPolicyActionReconciliation
Each policy evaluation snapshots the window and data it used, so an old decision does not silently change its explanation when traffic moves later.
Current implementation
Control planeFastAPI + SQLAlchemy + SQLite + Alembic
FrontendNext.js + TypeScript + Tailwind + Recharts
Model servingscikit-learn + joblib + FastAPI
RoutingWeighted router with static version → host mapping
Verificationpytest + Ruff + mypy --strict + Locust
RuntimeDocker + Docker Compose
Reference implementation with explicit limits
This is a production-oriented local control-plane reference implementation, not a production certification. The current scope intentionally uses SQLite, one router, Docker Compose and open demo endpoints.
Kubernetes, PostgreSQL at deployment scale, Kafka-backed metrics, MLflow, OIDC/RBAC, multi-router reconciliation and longer soak testing are documented production evolution steps, not claims about the current implementation.