ÖFK
Menu
All projects

Structured Data / Governed Text-to-SQL

Governed benchmark system

DecisionSQL

A governed one-shot Text-to-SQL system and execution-based benchmark where the model proposes a single typed decision and deterministic software owns SQL admission and read-only execution.

The model proposes. Deterministic software decides what may execute.
View repository Frozen synthetic governed benchmark, with documented limits
M48B.2 · frozen 90-case synthetic benchmark78 / 90 = 86.7%
Governed task successOne-shot governed decisions on a frozen synthetic governed benchmark. Not general Text-to-SQL accuracy or production accuracy.
Authority blocks held 15/150 / 15
Unauthorized answersEvery authority-blocked case that could not be answered through authorized relationships was refused; no unauthorized data was returned.
Given the model chose ANSWER51 / 53 = 96.2%
Conditional runtime correctnessOf the answerable cases where the model committed to ANSWER + SQL, runtime execution matched the result contract on the BASE state and across counterfactual states.
Execution-based semantic evaluation · 0 surviving190 / 190
Mutants killed120/120 reference witnesses and 184/184 counterfactual fixture comparisons; mutants are intentionally wrong behaviours used to prove the fixtures discriminate semantic errors.

A demo answers syntax; this answers whether the query may run

A model can produce SQL that is syntactically fine and still unauthorized, unbounded or semantically wrong. DecisionSQL keeps the model’s proposal and the authority to execute separate, and evaluates governance apart from query execution because some requests should refuse SQL entirely.

Correctness is decided by running the query against changing database states, not by comparing it to a reference string. The benchmark measures first-pass behavior under a one-shot contract, so model decisions and server enforcement both stay visible.

What the system does

DecisionSQL covers the path from a governed request to a bounded result: one typed decision, a deterministic admission chain, and execution-based scoring across counterfactual states.

Governed decision

  • One typed decision per request: answer, clarification or block
  • Model-visible governed context, evaluator-only truth kept separate
  • Authority and policy blocks that never reach the SQL runtime

Deterministic SQL admission

  • sqlglot parse, SQL / object / function policy
  • Server-owned grain safety with a narrow deterministic normalizer
  • PostgreSQL EXPLAIN, a frozen cost gate and an accepted QueryPlan

Execution-based evaluation

  • Two reference witnesses and a typed ResultContract
  • Counterfactual fixtures that make wrong semantics diverge
  • Restricted read-only execution under reader role and timeout

One submission, a deterministic admission chain

Only an ANSWER + SQL submission enters the SQL runtime. The selected SQL then crosses each stage below before a restricted reader runs it, and the executor never accepts SQL directly from the model, the normalizer or the evaluator.

  1. 01

    One typed decision

    The model receives the question plus a governed context and emits exactly one decision. Runtime routing follows the parsed submission, not evaluator truth, so only ANSWER + SQL enters the SQL runtime.

  2. 02

    Parse and policy

    sqlglot parses the selected SQL and the policy enforces one read-only statement, governed object access, function restrictions and complexity limits before anything else runs.

  3. 03

    Grain safety

    A GrainSafetyValidator detects parent-measure fanout, where a join to several child rows can silently multiply a SUM. Server-owned metadata, not the model, owns that contract.

  4. 04

    Narrow normalization, re-validated

    For the supported additive-parent, declared 1:N shape a deterministic normalizer preaggregates on the child side. Normalized SQL is never trusted automatically; it re-passes parse, policy and post-grain validation with no raw unsafe fallback.

  5. 05

    Cost admission

    PostgreSQL EXPLAIN runs on a deterministically analyzed state, and a frozen cost gate rejects SQL above the benchmark's max rows or max cost before execution.

  6. 06

    Accepted QueryPlan, then execution

    Execution requires an accepted immutable QueryPlan issued by the SQL safety service. A restricted reader runs it under a read-only transaction, reader role, statement timeout and bounded result rows.

Proposal and execution authority stay separate

The model owns the first-pass decision and, when it answers, the proposed SQL. It does not own authorization, physical schema truth, grain semantics or the right to execute. Deterministic software owns that half.

The model ownsDeterministic software owns
  • The first-pass typed decision
  • The proposed SQL when it chooses ANSWER
  • Nothing about authorization or execution
  • No physical schema truth or grain semantics
  • No right to run the query it wrote
  • Typed decision validation
  • SQL parsing, policy and server-owned semantic metadata
  • Narrow grain-safe normalization
  • PostgreSQL EXPLAIN, cost admission and QueryPlan creation
  • Restricted read-only execution and result comparison

Runtime routing follows the parsed submission, not evaluator truth. An ANSWER + SQL runs even when the truth is AMBIGUOUS; the harness records that runtime outcome separately instead of skipping the branch.

Architecture

A governed context feeds one typed decision. Only ANSWER + SQL enters the admission chain — parse, policy, grain safety, narrow normalization, EXPLAIN, a cost gate and an accepted QueryPlan — before restricted read-only execution. A separate evaluator scores the outcome off the request path against counterfactual states.

A natural-language analytics request is answered against a model-visible, governed context, and the model emits exactly one typed decision. Only an ANSWER + SQL submission enters the SQL runtime; clarifications and authority or policy blocks never do. The selected SQL then crosses a deterministic admission chain — sqlglot parse, SQL policy, grain-safety validation, an optional narrow normalizer, re-validation, PostgreSQL EXPLAIN, a cost gate and an accepted immutable QueryPlan — before a restricted read-only executor runs it. Correctness is scored off the request path by executing against BASE and counterfactual database states, with evaluator-only truth that never reaches the model.

Governed decision

Runtime routing is driven by the parsed model submission, not by evaluator truth; only ANSWER + SQL reaches the SQL runtime.

Analytics Request with governed context Model-Visible Governed Context. Model-Visible Governed Context one-shot generation One-Shot LLM Decision. One-Shot LLM Decision enters SQL runtime ANSWER + SQL. One-Shot LLM Decision governance outcome NEEDS_CLARIFICATION. One-Shot LLM Decision governance outcome BLOCKED_AUTHORITY. One-Shot LLM Decision governance outcome BLOCKED_POLICY.

  1. ClientAnalytics Requestnatural language
  2. ControlModel-Visible Governed Contextpublic / governed schema, authorized metadata
  3. AnalyzerOne-Shot LLM Decisionuntrusted, typed, no retry
  1. ServiceANSWER + SQLread-only SELECT
    OutputNEEDS_CLARIFICATIONno SQL runtime
    BoundaryBLOCKED_AUTHORITYno SQL runtime
    BoundaryBLOCKED_POLICYno SQL runtime

Runtime admission (ANSWER + SQL)

The executor never accepts SQL directly from the model, the normalizer or the evaluator; execution requires an accepted immutable QueryPlan.

Selected SQL parses sqlglot Parse. sqlglot Parse policy SQL / Object / Function Policy. SQL / Object / Function Policy checks grain GrainSafetyValidator. GrainSafetyValidator normalizes Grain-Safe Normalizer. Grain-Safe Normalizer re-validates Re-Parse + Re-Policy + Post-Grain. Re-Parse + Re-Policy + Post-Grain plans PostgreSQL EXPLAIN. PostgreSQL EXPLAIN cost gate Query Cost Gate. Query Cost Gate accepts Accepted QueryPlan. Accepted QueryPlan executes Restricted Read-Only Executor. Restricted Read-Only Executor returns Bounded Result.

  1. ServiceSelected SQLfrom the ANSWER submission
  2. Controlsqlglot Parseone statement, read-only
  3. ControlSQL / Object / Function Policygoverned access, complexity limits
  4. AnalyzerGrainSafetyValidatorparent-measure fanout detection
  1. ControlGrain-Safe Normalizernarrow supported shape, fail-closed
  2. ControlRe-Parse + Re-Policy + Post-Grainno raw unsafe fallback
  3. AnalyzerPostgreSQL EXPLAINplanner cost on analyzed state
  4. ControlQuery Cost Gatefrozen max rows / max cost
  1. PersistenceAccepted QueryPlanimmutable, issued by the SQL safety service
  2. ServiceRestricted Read-Only Executorreader role, statement timeout, bounded rows
  3. OutputBounded Resultrow-capped

Execution-based evaluation

Truth decides whether the governed decision is correct and whether result-contract evaluation applies; it does not decide whether SQL enters runtime.

Runtime + Governance Outcome scored against Execution-Based Evaluator. Execution-Based Evaluator uses Witnesses + Counterfactual Fixtures. Witnesses + Counterfactual Fixtures yields Governed Task Success.

  1. OutputRuntime + Governance Outcomerecorded per case
  2. AnalyzerExecution-Based Evaluatorevaluator-only truth, off the request path
  3. PersistenceWitnesses + Counterfactual Fixtures2 reference witnesses, typed ResultContract
  4. OutputGoverned Task Success78 / 90 on the frozen benchmark

Architecture notes

  • Runtime routing is submission-driven: an ANSWER + SQL enters the SQL runtime even when the truth is AMBIGUOUS, so a wrong governed decision still produces a typed outcome instead of a crash.
  • The grain-safe normalizer is intentionally narrow — additive parent measure, a declared 1:N relationship and a supported LEFT JOIN fanout shape — and stays fail-closed or non-target outside it. It is not a universal fanout solver.
  • PostgreSQL ANALYZE runs as environment preparation before reader planning, outside SqlSafetyService, the cost gate, the executor and the request path, so EXPLAIN costs are deterministic and excluded from request latency.
  • Reference SQL, fixtures and expected results are evaluator-only and never enter the model request; correctness is execution-based semantic correctness, not SQL string or AST equality.
  • These are results on a frozen 90-case synthetic governed benchmark under a documented one-shot contract, not general Text-to-SQL accuracy, production accuracy or universal SQL safety.

Four reliability decisions

Submission-driven runtime, truth-driven evaluation

Runtime routing is decided by the parsed model submission, not by evaluator truth. An ANSWER + SQL receives the full parse, policy, semantic, cost, QueryPlan and execution treatment even when the truth is AMBIGUOUS, and the harness records the runtime outcome separately instead of crashing on a wrong governed decision.

Server-owned grain safety

Joining a parent to several child rows can triple a parent measure. The model is not trusted to own that contract. Server-owned metadata describes entity and grain keys, relationship cardinality and aggregation behavior, and the normalizer stays fail-closed outside its frozen additive-parent, declared 1:N shape.

Execution-based correctness, not string match

Candidate SQL is not judged by exact text or AST equality. Two independent reference witnesses, a typed ResultContract and counterfactual fixtures that change distributions decide correctness, so SQL that accidentally returns the right rows on one state fails when the semantics actually diverge.

One-shot, with provenance discipline

One benchmark case yields one semantic attempt with no retry, repair, judge, selector or reflection, which keeps model decision errors and server enforcement observable. The corpus was assembled with 0 retries and 0 duplicate attempts under a frozen prompt and planner-statistics contract, with request-hash compatibility verified.

Server-owned grain safety, made concrete

Joining one parent row to several child rows can silently multiply a parent measure. The model is not trusted to own this contract; server-owned metadata and a narrow deterministic normalizer are.

Naive join

Fanout multiplies the measure

order #A1amount 100
line 1× 100
line 2× 100
line 3× 100
SUM(order.amount) = 300counted 3×
Grain-safe

Child-side preaggregation

lines → subtotalpreaggregated
order #A1joined once
SUM(order.amount) = 100counted once

Additive parent measure, a declared 1:N relationship and a supported LEFT JOIN fanout shape resolve to deterministic child-side preaggregation. Outside that shape the normalizer stays fail-closed, and normalized SQL still re-passes parse, policy, EXPLAIN and the cost gate before it can run.

4 / 4 fanout states normalized100% precision0 regressions0 unsafe raw fallback

Governance is scored apart from execution

The benchmark includes cases where producing SQL is the wrong behavior, so refusing is a measured outcome, not a failure. The four behaviors are scored separately rather than merged into one accuracy number.

BehaviorCasesExpected decisionResult
ANSWERABLE60ANSWER + one read-only SELECT51 / 60 delivered correct
AUTHORITY_BLOCKED15BLOCKED_AUTHORITY15 / 15 · 0 unauthorized answers
AMBIGUOUS9NEEDS_CLARIFICATION6 / 9
POLICY_BLOCKED6BLOCKED_POLICY6 / 6

The three governance residuals are all in the ambiguity category; authority held at 15 / 15 with no unauthorized answers. These are frozen synthetic benchmark results, not production accuracy.

Selected benchmark evidence

Governed decisions, runtime admission, grain safety and semantic discrimination describe different failure surfaces. They stay separate instead of collapsing into one project score.

Evidence sliceCurrent resultWhat it covers
Governed task success78 / 90 = 86.7%One-shot governed decisions across all four behaviors on the frozen synthetic benchmark. Answerable end-to-end runtime task success accuracy was 51 / 60 = 85.0%.
Runtime admission0 rejections · 0 execution failuresAcross the 53 answerable cases where the model chose ANSWER, every submission passed parse, policy, semantic admission, cost and execution. 2 result mismatches remained, giving 51 / 53 = 96.2% conditional runtime correctness.
Grain normalization4 / 4 · 100% precisionEvery PARENT_MEASURE_FANOUT state was normalized with 0 regressions, 0 unauthorized relationships introduced and 0 unsafe raw fallback, inside the supported shape only.
Semantic discrimination190 / 190 mutants killed120 / 120 reference witnesses and 184 / 184 counterfactual fixture comparisons with 0 invalid and 0 surviving mutants, proving the fixtures actually separate semantic errors.
Branch-complete harness360 / 360 scenarios90 cases across 4 valid decisions exercised 90 ANSWER runtime routes and 270 non-ANSWER bypasses, verified before the remaining responses were generated.

Current implementation

RuntimePython 3.12 + FastAPI + PostgreSQL
SQL safetysqlglot parse + policy + grain safety + cost gate
ExecutionAccepted QueryPlan + restricted read-only reader
DataSQLAlchemy + Alembic + synthetic enterprise packs
EvaluationCounterfactual fixtures + reference witnesses + mutation testing
Qualitypytest + Ruff + mypy + OpenTelemetry

Governed benchmark with explicit limits

The repository is a governed one-shot benchmark and reference runtime. Its numbers belong to the frozen contracts, synthetic packs and one-shot conditions they were measured under.

Limits documented
  • The evidence supports governed one-shot decision evaluation, deterministic SQL safety, the narrow grain-normalization mechanism, an accepted-QueryPlan execution boundary and reproducible planner state.
  • It does not establish universal Text-to-SQL correctness, universal fanout or grain repair, or production readiness for arbitrary enterprise schemas.
  • The SQL policy is a deterministic application boundary, not complete tenant-level authorization or RLS coverage.
  • Numbers belong to the frozen runtime and benchmark contracts, on synthetic enterprise-style packs, not to production traffic.

Deep dive

Repository README and walkthroughM48B.2 end-to-end summaryMachine-readable summaryBranch-complete runtime contract
Open the full repository