ML Platform

Current validation, with explicit boundaries

Reviewed against main at 6d4e9a2 (8 October 2026). These are distinct acceptance checks on a single-node ARM64 kind laboratory, not a unified benchmark or customer deployment. The committed reports record methods, identities and artifact hashes; some raw data and logs remain in external archives.

NYC TLC, local historical-data replay, 8 Oct 20261,000,000

Trips scored and matched to labels100% feedback coverage, zero unmatched rows.

Single node, warm images, Argo overhead included18.95 s

Scheduled scoring pipeline1 CPU / 1 GiB main container; separate monitoring Job took 28.17 s. One recorded execution, not sustained throughput.

Go gateway, final-image acceptance, 7 Oct 2026200/200

Function responses during rolling restartReal function requests with validated IDs and headers. Availability probes, not load-capacity measurement.

Monitoring automation, local artifact, 8 Oct 202612/12

Control-plane release checksMigration 0030, dependency isolation, readiness recovery, SBOM and severity/secrets scans.

From a processing date to an inspectable quality report

The NYC TLC case study uses one million cleaned, sampled taxi trips from each of January and February 2025. A fare model trains locally on 200,000 January rows, then enters MLflow with a frozen SHA-256 manifest. The platform owns the subsequent scheduled scoring and monitoring workflow.

1. 01

   Freeze the execution inputs

   The schedule resolves processing date 2025-02-01 to taxi-features v2. Typed parameters, model and dataset identities persist in an immutable execution snapshot before Argo starts work.

2. 02

   Score within explicit resource bounds

   The worker verifies input integrity and the model manifest, then scores one million rows in 10,000-row batches. The recorded pipeline takes 18.95 s including Argo overhead, with a 1 CPU / 1 GiB main container and warm images.

3. 03

   Publish versioned predictions and lineage

   Outputs carry their producer run, pipeline step, input dataset and model identity. An independent rerun publishes v2 at a separate S3 location with identical bytes; replaying its creation key returns the same run.

4. 04

   Measure quality against recorded fares

   A separate monitoring Job joins every prediction by trip_id: 100% coverage, zero unmatched rows. MAE 2.082 USD, RMSE 4.466 USD and R² 0.920 agree with independent pandas/sklearn calculations. The Job takes 28.17 s in the recorded run.

5. 05

   Trigger future checks from publication

   A follow-up run publishes v3 and automatically creates monitoring. Publication and its outbox event commit together; a unique rule/dataset intent survives reconciler restart. Feedback is date-matched and frozen, or the execution waits within its deadline.

All four selected features were STABLE at PSI threshold 0.2 (PSI 0.000405–0.007974). That describes these cleaned windows; no drift was injected and STABLE does not rule out every quality problem. Ground truth was withheld before scoring, not collected from live inference. The demonstration schedule is paused.

One platform intent per dataset occurrence is not exactly-once external side effects. Monitoring failure is independent of producer success. Same-DAG dynamic output binding, rolling monitoring windows, online prediction capture and a Feature Store remain outside the demonstrated scope. Read the automation contract.

The web workspace

Current repository captures show schedules, versioned datasets, batch definitions and model-quality reports. The Credit Risk scenario uses synthetic demo data; these screens are separate from the real NYC acceptance evidence above.

From training to a governed endpoint

System architecture

The control plane records intent and returns; background reconciliation handles infrastructure. The data plane checks caller access and endpoint state before forwarding traffic. External systems sit behind adapters, keeping lifecycle rules independent of their SDKs. The API/reconcilers remain Python; the chart defaults to a separate Go gateway with a Python rollback option. The CPU path is lab-verified; the vLLM path below represents implemented contracts awaiting GPU acceptance.

A model moves through explicit states

1. 01

   Create a project and establish access

   The control plane records the project and memberships; the project reconciler targets a dedicated namespace with quotas, limits and network policy.

2. 02

   Run a training job or pipeline

   A job or DAG becomes desired state. Reconcilers submit Argo workflows and bring run status, steps and failure reasons back into the platform.

3. 03

   Evaluate and promote a version

   Register an MLflow artifact or Hugging Face version, evaluate it against acceptance thresholds, then promote an eligible candidate. Model-alias reconciliation aligns the registry with platform state.

4. 04

   Create an immutable serving revision

   An authorized request checks version eligibility, runtime kind and GPU quota, then commits the revision, desired deployment state and audit context to PostgreSQL.

5. 05

   Reconcile and gate the rollout

   Deployment reconciliation targets KServe readiness. The rollout loop shifts canary traffic, reads revision metrics and advances or rolls back according to its gates.

6. 06

   Expose an endpoint through the gateway

   Callers use scoped API keys or OIDC tokens. The separate gateway resolves ready endpoints, applies limits, forwards or streams the response and records usage.

Registration, evaluation, promotion, deployment readiness and canary success are separate decisions with their own state transitions.

Access follows the project; traffic follows its plane

OIDC supplies identity. Project membership and a fail-closed policy table govern management operations; the gateway applies caller scope to exposed endpoints. Namespace policy and quotas define the workload boundary behind those application rules.

Design decisions

Infrastructure foundation

The platform grew from a local Kubernetes inference service with MLflow, PostgreSQL, MinIO, GitOps and observability. The measurements below belong to that original infrastructure: they describe its load, autoscaling and recovery behavior, separately from the broader platform design.

Historical measurements remain scoped to the original service; the current Go gateway and expanded platform do not inherit its throughput. Original milestone record.

Selected failure drills from the first infrastructure

Platform stack

Deployment

The control-plane Helm chart is deployed and runtime-verified on the local ARM64 lab. It defaults to two API, gateway and reconciler replicas, with PDBs and Lease-elected reconciliation. The AWS target is designed alongside it, backed by Terraform.

In short

What is ML Platform?

ML Platform joins training, scheduled batch inference, immutable dataset versions, recorded lineage, model-quality reports and serving in one project-scoped workspace. A Python API and reconcilers persist intent in PostgreSQL; a separate Go gateway governs inference traffic. Local Kubernetes acceptance includes the CPU lifecycle and a million-row historical NYC TLC scoring/monitoring workflow.

What problem does ML Platform solve?

Training a model is only one step in operating it. Teams also need ownership, evaluation rules, release decisions, serving access and a way to recover interrupted infrastructure changes. This platform organizes those concerns around durable lifecycle intent and distinct control and data planes, extending the original Kubernetes inference foundation into a shared ML workspace.

Related

Deep dive