Agentic SRE is a Kubernetes incident root-cause analysis system that separates the investigator from the judge. A bounded state machine selects one legal, read-only observation at a time; every observation is normalized into a typed Finding before it can move a hypothesis, and deterministic verification and resolution own the root cause. An LLM is optional and the measured benchmark path uses zero model calls; the model can choose among already-legal reads but can never create evidence, findings, hypotheses or the final diagnosis.

What problem does Agentic SRE solve?

An incident root-cause tool becomes a trust problem the moment it can act on a cluster or hand an operator an unverified answer. A plausible model guess is not a diagnosis. Agentic SRE keeps evidence acquisition and root-cause judgment separate: the investigator is bounded and read-only, the judge is deterministic and auditable, and the whole architecture was frozen and evaluated on a blind holdout whose predictions were hashed before grading — so accuracy, calibration and safety are measured as distinct evidence rather than asserted as one headline.

All projects

Agent Systems / AI Platform

Agentic SRE

An evidence-driven root-cause analysis engine for Kubernetes incidents. A bounded, read-only investigator gathers evidence over changes, events, logs, traces, dependencies and topology, while a deterministic RCA engine — not the model — makes the final root-cause judgment.

The investigator gathers evidence. It does not decide the root cause.

View repository Frozen blind ITBench-Lite holdout · deterministic path · 0 model calls

Frozen ITBench-Lite TEST25 holdout21/25 (84%): Blind exact-root agreementAll 25 bounded predictions were persisted and SHA256-hashed before any full-source diagnosis was graded; only an exact canonical root entity counts. Synthetic 25-scenario benchmark, not production accuracy.
Deterministic investigation policy0 MODEL CALLS: Measured benchmark pathThe graded run made zero model calls. An LLM is optional and can only choose among already-legal reads; it never owns the diagnosis.
TEST25 · LIKELY 11/12 · UNVERIFIED 1/49/9 VERIFIED: Confidence calibrationConfidence tiers are reported honestly rather than collapsed into one accuracy figure; NO_DATA is treated as neutral, not as evidence for a theory.
6 validated reads/incident across 25 incidents150 READS · 0 ERRORS: Bounded investigationOne read at a time from a legal observation surface: 2,226 new evidence references normalized into 244 typed Findings, with 0 tool errors.

An LLM can guess. Only evidence can diagnose.

The engine does not ask a model what caused an incident. It acquires bounded, read-only evidence, normalizes that evidence into typed Findings, and rebuilds a deterministic diagnosis. An LLM is optional and the measured benchmark path used zero model calls; the model may choose among already-legal reads but can never create evidence, Findings, hypotheses or the final root cause.

Accuracy, calibration and safety are measured as distinct evidence: 21/25 exact-root agreement on a frozen blind holdout is reported separately from confidence calibration and from the read-only trust boundary, rather than compressed into one score.

What the system does

A deterministic RCA engine, a bounded investigation runtime around it, and the observation and control plane that feed and surface an incident.

Deterministic RCA engine

Typed causal signals over changes, events, dependencies and topology
Hypothesis rebuild, verification, confidence and resolution owned in code
Root-entity selection with an explicit causal path, not an opaque answer

Bounded investigation runtime

One validated read at a time from a legal observation surface
Explicit turn, tool, wall-time, per-gap and no-progress limits
Optional LLM policy chooses among already-legal reads — never creates evidence

Observation and control plane

Kubernetes object versions and Events, Alertmanager, Loki, traces, snapshots
Incident lifecycle, persistence, API, CLI and HTML/UI reporting
Read-only access; remediation is proposed for an operator, never executed

One observation, normalized before it can move a diagnosis

Investigation is a controlled loop, not an open-ended chat. Every new observation returns through the same deterministic path before it is allowed to change anything.

01
Alert and observation cutoff
Investigation starts from an alert and an explicit observation cutoff, reconstructing the changes and symptoms visible at the incident boundary.
02
Deterministic RCA forms hypotheses
The engine identifies candidate causal actors and open information gaps from typed Findings — before any bounded read is spent.
03
Investigator selects one legal read
A bounded state machine picks a single validated, read-only observation for the most valuable gap, checks scope and budget, then executes exactly one read.
04
Evidence becomes a typed Finding
The observation enters the EvidenceStore and is normalized into a typed Finding; nothing can move a diagnosis until it has crossed this boundary.
05
Hypotheses rebuilt, then verified
The same deterministic RCA and normalization code rebuilds hypotheses and re-runs verification and resolution — remaining gaps loop back for another bounded read.
06
Root cause, confidence and proposal
The run finalizes a root entity, confidence, resolution state, evidence and causal path, plus a proposed remediation that is returned to an operator and never executed.

observation → EvidenceStore → normalization → Finding → hypothesis rebuild → verification → resolution. The same RCA and normalization code runs for initial observations and new investigation evidence.

A read-only trust boundary, judgment kept out of the model

The investigator can acquire evidence but never owns the diagnosis, and the whole system can observe a cluster but never change it. Both boundaries are explicit.

Investigator may (read-only)	Deterministic judge owns · model never does
select one legal, in-scope observation at a time read Kubernetes object versions, Events and configured snapshots read bounded Loki logs, traces and Alertmanager context an optional LLM may choose among already-legal reads	normalizing observations into typed Findings rebuilding hypotheses, verification, confidence and resolution selecting the root entity and its causal path Secrets are not read; remediation is proposed, never executed

Kubernetes observation is read-only and Secrets are deliberately not read. Invalid actions, duplicate reads, tool errors, NO_DATA and exhausted budgets terminate safely with the current deterministic diagnosis rather than forcing an answer.

Architecture

Deterministic RCA sits at the center: it forms hypotheses and information gaps, the bounded investigator spends one legal read at a time, and every observation is normalized into a Finding before hypotheses are rebuilt and resolved. Kubernetes access stays read-only and remediation is proposed, never executed.

Deterministic RCA loop

The engine forms hypotheses, the bounded investigator spends one legal read, and evidence is normalized into a Finding before the diagnosis can change.

ClientAlert + observation cutoffchanges and symptoms at the boundary
ControlDeterministic RCAhypotheses · information gaps

AnalyzerBounded investigatorselect · validate scope + budget
ServiceOne legal observationturn · tool · wall-time · per-gap limits

ControlEvidenceStore → normalizationtyped Finding
OutputRoot cause + causal pathconfidence · resolution · evidence
BoundaryRemaining gaploops back for one more read

Observation sources

Bounded, read-only observation over the incident's telemetry; Secrets are deliberately not read.

ObservabilityKubernetes objects + Eventsversions · warnings
ObservabilityAlertmanagerincident context
ObservabilityLoki logs · tracesbounded, replayable
PersistenceSnapshot dataconfigured sources

ControlLegal observation surfaceallowlisted capabilities only
BoundarySecretsdeliberately not read

Control plane and remediation

Incident lifecycle, persistence and reporting; remediation is proposed for an operator and never executed.

ServiceControl planeFastAPI · lifecycle · persistence
OutputCLI · HTML/UI reportroot cause · evidence · causal path
BoundaryProposed remediationoperator reviews · never auto-executed

Architecture notes

The investigator and the judge are separate: an optional LLM policy can choose among already-legal reads, but deterministic normalization, hypothesis rebuilding, verification and root-cause resolution stay authoritative, and the measured benchmark path used zero model calls.
Every observation is normalized into a typed Finding before it can move a hypothesis; on the frozen TEST25 run, 2,226 new evidence references became 244 Findings across 150 bounded reads with zero tool errors.
NO_DATA is treated as neutral rather than evidence for a theory, and invalid actions, duplicate reads, tool errors and exhausted budgets terminate safely with the current deterministic diagnosis.
Kubernetes access is read-only and Secrets are deliberately not read; remediation is returned as a proposal for an operator and is never executed, with no arbitrary shell or cluster-write path.
The blind TEST25 result (21/25 exact-root, 0 model calls) is evidence on a pinned 25-scenario ITBench-Lite set with predictions hashed before grading, not a universal production accuracy guarantee.

Engineering decisions that keep the diagnosis honest

The investigator and the judge are separate

An LLM can turn a plausible answer into an unverified diagnosis, so the model is confined to selecting already-legal reads. Deterministic normalization, hypothesis rebuilding, verification and root-cause resolution stay authoritative, and the measured benchmark path runs with zero model calls.

Evidence is typed before it can move a hypothesis

A raw observation should not silently shift a root cause. Every observation flows through EvidenceStore → normalization → typed Finding before hypotheses are rebuilt; on the frozen run 2,226 new evidence references became 244 Findings along an auditable path.

NO_DATA is neutral, not evidence for a theory

A missing signal is reported as missing rather than counted against a hypothesis. The frozen run recorded 95 NO_DATA observations that were held as neutral, keeping absent telemetry from being turned into false support.

Causal topology, not graph proximity

Being near a symptom in a graph does not make an entity the cause. Ownership, configuration use, declared dependencies, policies, fault targets, scaling relationships and workload topology are interpreted as explicit directional relations, so a diagnosis carries a real causal path.

The architecture was frozen before it was graded

A benchmark the system was tuned against measures fit, not capability. The architecture was frozen at a pinned commit, TEST25 was held out, and all 25 bounded predictions were persisted and SHA256-hashed before any full-source diagnosis was opened — no code changed after holdout results were visible.

Calibration, not just accuracy

A confidently wrong diagnosis is worse than an honest abstention, so confidence tiers are reported as measured on the blind TEST25 run rather than collapsed into one accuracy figure.

Confidence	Correct	Total
VERIFIED	9	9
LIKELY	11	12
UNVERIFIED	1	4

NO_DATA is treated as neutral rather than proof against a hypothesis; the graded run recorded 95 NO_DATA observations that were held as neutral instead of forced into a theory.

Selected benchmark evidence

Blind accuracy, the development split, calibration, bounded investigation and the trust boundary describe different claims. They stay separate instead of being compressed into one number.

Evidence slice	Current result	What it means
Blind TEST25 holdout	21/25 (84%) exact root · 0 model calls	Predictions persisted and SHA256-hashed before grading; only an exact canonical root entity counts. Synthetic 25-scenario benchmark, not production accuracy.
Development split	10/10 exact root	DEV10 is the split used while building the frozen architecture, reported apart from the blind holdout so development evidence is never confused with it.
Combined	31/35 (88.6%)	DEV10 + TEST25 together, kept as a separate line rather than a headline that hides the blind-vs-development distinction.
Confidence calibration	VERIFIED 9/9 · LIKELY 11/12 · UNVERIFIED 1/4	Confidence tiers carry measured meaning; a confident label is not decorative and NO_DATA stays neutral.
Bounded investigation	150 reads · 0 tool errors	Six validated reads per incident across 25 incidents: 2,226 new evidence references, 244 normalized Findings, 37 decision-relevant calls, 95 NO_DATA observations.
Trust boundary	0 cluster writes	Read-only observation with Secrets excluded and allowlisted capabilities only; remediation is proposed but never executed, and there is no arbitrary shell path.

A blind holdout, hashed before grading

TEST25 was held out until the architecture was frozen. For the holdout, all 25 bounded predictions were persisted and SHA256-hashed before any full-source diagnosis was opened, and grading then compared exact canonical root entities by scenario ID — a same-workload or nearby entity does not count as a match.

The run used the pinned ITBench-Lite revision d0916b0 with manifest 08a5e56, the architecture was frozen at commit 8ccce16, and the prediction artifact was hashed (879cab5…), so the 21/25 result is reproducible rather than a one-off claim. It is evidence on this pinned 25-scenario set, not a universal production accuracy guarantee.

Current implementation

RCA engineDeterministic signals · causal topology · verification · resolution

Investigation runtimeBounded LangGraph state machine · validated read-only tools

Observation sourcesKubernetes objects + Events · Alertmanager · Loki · traces · snapshots

Control planeFastAPI + SQLAlchemy + Alembic + PostgreSQL · CLI · HTML/UI reporting

Live validationkind + Prometheus + Alertmanager + Chaos Mesh lifecycle gate

BenchmarkITBench-Lite · frozen revision + manifest · hashed predictions

Quality gatesRuff · Mypy · pytest · pre-commit · OpenTelemetry

Controlled evaluation, with explicit limits

Agentic SRE is suitable for controlled evaluation and read-only incident-assistance workflows, with a real kind lifecycle gate and a frozen blind benchmark. It is not a universal replacement for an experienced SRE.

Current generalization evidence is the frozen 25-scenario blind TEST25 run; larger and more diverse production datasets are still needed. 84% is a measured benchmark result, not a universal accuracy guarantee.
The diagnosis is deterministic, but bounded query windows and captured telemetry can miss older or unavailable decisive evidence.
Captured Loki data is replayable evidence, not a complete historical log archive, and investigation runs under fixed turn, read, wall-time and per-gap budgets.
Some diagnoses depend on the configured read APIs and their authentication; built-in read endpoints are unauthenticated by default and are an operator responsibility.
The supported deployment is single-process / single-replica rather than highly available.
It is evidence-driven RCA, not formal causal inference, and there is no autonomous remediation, arbitrary shell execution or cluster write capability.

Deep dive

Architecture details Evaluation methodology Frozen benchmark report Architecture decision records Release documentation

Open the full repository