ÖFK
Menu
All projects

AI Infrastructure / Platform Engineering

ML Platform Infrastructure

A local ML platform reference implementation on Kubernetes: an inference service, its full MLflow/PostgreSQL/MinIO lifecycle, GitOps, autoscaling, security hardening and observability, validated with real failure drills instead of descriptions.

View repository local-v1.0.0 · Kubernetes implementation frozen · AWS design-only, not yet applied
k6 · saturated 2,935 req/s645,809 REQUESTS
Load test, 0% errorsSaturated /predict p95 latency was 32.9 ms on the local kind cluster under k6 load.
Scale-down 6 → 2 in ~230s, stepped2 → 6 IN 71s
HPA scale-up under loadHorizontal Pod Autoscaler behavior measured end-to-end against real CPU-driven load, not simulated.
Deleted pod replacement · Argo CD self-heal12–15s / ~1.4s
Pod recovery / drift reconciliationA deleted pod has a replacement serving in 12–15s. Argo CD detects and reconciles config drift in about 1.4 seconds, independent of its own ~3 min Git poll.
M11 · cluster, images, build cache destroyed first11/11 IN 901s
Reproducibility from scratchmake local-up then make local-test passed every acceptance check after a full teardown, proving the repository alone is sufficient, not a pre-warmed environment.

A diagram is a claim. A drill is evidence.

Every headline number here comes from running the cluster and breaking it on purpose, not from documenting intent: GitOps reconciliation speed, autoscaling under load, pod recovery time, a NetworkPolicy deny rule, and eight injected faults were all measured against a live kind cluster.

AWS is designed as code — Terraform, a cost model, a migration doc — but is explicitly marked not-yet-applied, so local evidence is never mistaken for a cloud deployment claim.

What the platform does

An inference service, its full ML lifecycle, and the GitOps, autoscaling, security and observability layers around it, all running inside one Kubernetes cluster.

GitOps deployment

  • Git is the source of truth: Helm charts, GitOps manifests, Terraform
  • Argo CD watches live cluster state continuously, drift reverted in ~1.4s
  • Argo CD polls Git independently on a ~3 min default cadence

ML lifecycle on Kubernetes

  • Inference service behind a Kubernetes Service, HPA 2↔6 on CPU
  • MLflow tracking backed by PostgreSQL and MinIO, both StatefulSets with PVCs
  • Pod Disruption Budget minAvailable=1 keeps the service up during voluntary disruption

Security and observability

  • NetworkPolicy default-deny plus an explicit allow-list, verified not assumed
  • Pod Security Standards: restricted, enforced cluster-wide
  • Prometheus scrapes /metrics; Grafana dashboards; Alertmanager with 5 promtool-tested rules

Two reconciliation loops, running at different speeds

Argo CD watches live cluster state continuously and also polls Git independently. The two paths have very different latency, which is why a Git commit lands slower than a manual edit gets reverted.

  1. 01

    Fresh cluster, nothing pre-existing

    make local-up builds a kind cluster, bootstraps GitOps, runs the ML lifecycle and stands up observability from a clean checkout, in roughly 15 minutes.

  2. 02

    Argo CD applies from Git

    Applications platform-local and inference-local are reconciled from helm/, gitops/ and infra/terraform/ — no manual kubectl apply.

  3. 03

    Two independent reconciliation loops

    Argo CD watches live cluster state continuously and self-heals in ~1.4s; it polls Git on its own ~3 min cadence. A manual edit is reverted almost instantly; a Git commit lands on the next poll.

  4. 04

    Traffic reaches a governed boundary

    The inference Service is the only path in. NetworkPolicy denies inference → PostgreSQL directly; the allowed path is inference → MLflow → PostgreSQL / MinIO.

  5. 05

    Load, faults and recovery are measured, not assumed

    k6 load tests, HPA scale events, pod deletion, Argo drift, and 8 injected faults are run against the live cluster and timed.

  6. 06

    make local-test closes the loop

    An 11-check acceptance suite runs against the live cluster; M11 proved the whole sequence again from a destroyed cluster, images and build cache.

Argo CD’s live-state watch self-heals drift in about 1.4 seconds; its independent Git poll defaults to roughly every 3 minutes.

A NetworkPolicy deny rule, verified rather than assumed

A default-deny NetworkPolicy plus an explicit allow-list governs which pods may reach which services. The inference → PostgreSQL direct path was actually attempted against the running cluster and confirmed denied, not inferred from the policy file.

AllowedDenied and verified
  • inference → MLflow (tracking calls)
  • MLflow → PostgreSQL (metadata)
  • MLflow → MinIO (artifacts)
  • Prometheus → inference /metrics (scrape)
  • inference → PostgreSQL directly
  • any pod outside the allow-list → ml-platform namespace
  • any workload outside Pod Security Standards: restricted
  • unauthenticated /predict traffic (Service boundary only)

Pod Security Standards: restricted is enforced cluster-wide. On its first run the security drill rejected its own probe pods under that same policy, which was initially misread as a NetworkPolicy denial rather than a Pod Security Standards rejection.

Architecture

Git is the only source of truth; Argo CD applies it into the cluster. Client traffic reaches only the inference Service, governed by an HPA and a PodDisruptionBudget. A default-deny NetworkPolicy allows inference → MLflow → PostgreSQL/MinIO while denying inference from reaching PostgreSQL directly. Prometheus feeds Grafana and a promtool-tested Alertmanager.

Git is the only source of truth: Helm charts, GitOps manifests and Terraform for the two Argo CD applications, platform-local and inference-local. Argo CD watches live cluster state continuously and self-heals drift in about 1.4 seconds, independent of its own roughly 3-minute Git poll. Client traffic reaches only the inference Service, which is fronted by an HPA scaling 2 to 6 replicas on CPU and a PodDisruptionBudget of minAvailable=1. A default-deny NetworkPolicy allows inference to call MLflow, and MLflow to reach PostgreSQL and MinIO, but denies inference from reaching PostgreSQL directly — verified by test, not assumed from the policy file. Prometheus scrapes the inference service and feeds Grafana and a 5-rule, promtool-tested Alertmanager.

GitOps deployment

Git is the source of truth; Argo CD applies it, watching live state continuously while polling Git on its own slower cadence.

Git poll ~3 min Argo CD. Argo CD watch + self-heal ~1.4s kind cluster. kind cluster applies Two independent loops.

  1. PersistenceGithelm/ · gitops/ · infra/terraform/
  2. ControlArgo CDplatform-local · inference-local
  3. Servicekind clusterPod Security Standards: restricted
  1. ObservabilityTwo independent loopslive watch ~1.4s vs Git poll ~3 min

Inference traffic

The Service is the only entry point; HPA and a PodDisruptionBudget govern how the Deployment absorbs load and voluntary disruption.

Client GET/POST inference Service. inference Service routes inference Deployment.

  1. ClientClientk6 load test · requests
  2. Serviceinference Service/health · /ready · /predict · /metrics
  3. Serviceinference DeploymentHPA 2↔6 on CPU · PDB minAvailable=1

ML lifecycle

NetworkPolicy default-deny allows inference → MLflow → PostgreSQL/MinIO, and denies inference from reaching PostgreSQL directly.

inference allowed MLflow. MLflow allowed PostgreSQL (StatefulSet + PVC). MLflow allowed MinIO (StatefulSet + PVC). PostgreSQL and MinIO then connects to inference → PostgreSQL directly.

  1. Serviceinferenceallowed
  2. ServiceMLflowtracking
  1. PersistencePostgreSQLStatefulSet + PVC
    PersistenceMinIOStatefulSet + PVC
  2. Boundaryinference → PostgreSQL directlydenied, verified by test

Observability

Metrics feed both a dashboard and an alerting path with rules validated by promtool before deployment.

inference /metrics scrapes Prometheus. Prometheus feeds Grafana (dashboards). Prometheus feeds Alertmanager (static thresholds).

  1. Serviceinference /metricsscraped
  2. ObservabilityPrometheus5 rules, promtool-tested
  3. OutputGrafanadashboards
    OutputAlertmanagerstatic thresholds

Architecture notes

  • Argo CD's two reconciliation paths run at very different speeds: continuous live-state watching self-heals drift in about 1.4 seconds, while its independent Git poll defaults to roughly every 3 minutes — a Git commit lands slower than a manual edit gets reverted.
  • NetworkPolicy denying inference from reaching PostgreSQL directly was verified by test against the running cluster, not assumed from the policy YAML, matching the project's stated preference for drills over descriptions.
  • PostgreSQL and MinIO run single-replica by design — an intentionally non-HA local lab, not a production high-availability claim.
  • Terraform for AWS (M13+) is at the design and static-validation level only (fmt, validate, tflint); no plan or apply has been run against an AWS account, and no cloud resource has been created.
  • Docker Compose was retired once the full local Kubernetes platform was reached; everything in this diagram runs inside the kind cluster.

Defects found by running automation, not written around

Blocking startup hid behind a slow dependency

Model loading ran on the request path at startup, so a slow artifact store took /health down with it. Fixed by moving model loading to a background thread so liveness and readiness stop being hostage to one dependency's latency.

A green rollout still dropped a request

kubectl rollout status reported success while an external probe measured 1 failure in 90 during a rolling update. The Kubernetes-level signal and the client-observed signal were both correct and still disagreed. Fixed with a preStop hook that drains in-flight connections before the pod terminates.

Zero errors and not instrumented looked identical

A labelled Prometheus counter emits no series until its first increment, so an empty dashboard panel could mean either. Fixed twice with or vector(0), because the first fix did not cover every label combination.

The security drill flagged itself

Pod Security Standards: restricted rejected the drill's own probe pods, and every admission rejection was misread as a NetworkPolicy DENY during the first run. Fixed by making the probe pods themselves PSS-compliant, then rerunning the drill against the real boundary.

A memory floor traded against CVEs

The 297 MiB MLflow image carried 7 unpatched CRITICAL CVEs; every fix landed only in a build with a 1.46 GiB floor. The tradeoff was made explicitly and the larger image was kept, rather than shipping the smaller image with known criticals.

Eight faults injected against the running cluster

Not simulated: each scenario below was actually triggered against the live cluster, with detection and recovery observed and timed.

ScenarioObservedRecovery
Pod crashReplicaSet notices; the surviving replica keeps servingreplacement ready in 12–15s
Invalid model artifactreadiness probe returns 503; pod held out of Service endpointsGit revert
Artifact store outage100% of requests still return 200 while the pod is not readybackground recheck, 0 restarts
Config driftArgo CD marks the app OutOfSync the moment it divergesself-heal in ~1.4s
Bad rolloutmaxUnavailable: 0 keeps old replicas servingGit revert, bad ReplicaSet pruned
Node drain with a stateful pod on itsurviving inference pod absorbs trafficPostgreSQL / MinIO reschedule automatically

Selected platform evidence

Load, autoscaling, recovery, reproducibility, failure engineering and the security boundary describe different parts of the system. They stay separate instead of being compressed into one score.

Evidence sliceCurrent resultWhat it means
Load test645,809 requests · 0% errorsk6 against the live cluster; saturated throughput 2,935 req/s at a saturated /predict p95 of 32.9 ms.
Autoscaling2 → 6 replicas in 71sHPA scale-up under load; scale-down back to 2 takes ~230s, stepped rather than immediate.
Recovery timing12–15s pod · ~1.4s driftPod deleted to replacement serving is 12–15s. Argo CD detects and reconciles config drift in ~1.4s, independent of its ~3 min Git poll.
Reproducibility11/11 acceptance · 901sCluster, images and build cache destroyed first, then make local-up → make local-test from the repo alone, proven in M11.
Failure engineering8 faults injected, not simulatedRun against the live cluster with measured detection and recovery, not described in documentation.
Security boundaryNetworkPolicy deny verifiedinference → PostgreSQL directly is denied and confirmed by test, not assumed from the policy YAML; Pod Security Standards: restricted is enforced.

Current implementation

OrchestrationKubernetes (kind) + Helm + Argo CD
ML lifecycleMLflow + PostgreSQL + MinIO, StatefulSets with PVCs
InferenceFastAPI inference service, HPA 2↔6, PDB minAvailable=1
ObservabilityPrometheus + Grafana + Alertmanager, 5 promtool-tested rules
SecurityNetworkPolicy default-deny + Pod Security Standards: restricted
IaC (design-only)Terraform: fmt / validate / tflint, no plan / apply yet
Load / verificationk6 load testing + make local-test 11-check acceptance suite

Local reference implementation with explicit limits

This is a validated local Kubernetes reference implementation, not a production certification and not a cloud deployment. AWS is designed as code and has not been applied.

  • Status is local-v1.0.0: the local Kubernetes implementation (M0–M12) is validated and frozen. AWS (M13+) has not started — no cloud resource has been created.
  • Terraform is at the design and static-validation level only (fmt, validate, tflint). No terraform plan or apply has been run against an AWS account.
  • PostgreSQL and MinIO run single-replica by design; this is an intentionally non-HA local lab, not a claim of production high availability.
  • Alerting uses static thresholds; there is no burn-rate or error-budget alerting yet.
  • make local-up and make local-test were proven manually from a destroyed-and-rebuilt environment, but do not yet run automatically in CI.
  • All committed credentials are disposable local-development defaults (for example MinIO's own upstream minioadmin/minioadmin). No production credentials, cloud secrets or customer data are included.

Deep dive

Milestone roadmapArchitecture notesFailure engineeringAWS migration designM0–M12 evidence transcripts
Open the full repository