ÖFK
Menu
All projects

Structured Data / Governed Text-to-SQL

Governed benchmark system

DecisionSQL

A governed one-shot Text-to-SQL system and execution-based benchmark where the model proposes a single typed decision and deterministic software owns SQL admission and read-only execution.

The model proposes. Deterministic software decides what may execute.
View repository Frozen synthetic governed benchmark, with documented limits
88.89%, 180-case governed benchmark160 / 180
Governed task successOne-shot governed decisions across 12 synthetic domains.
93.33%, model governance28 / 30
AuthorityTwo cases where the model wrongly chose ANSWER on an authority-blocked request; request-scoped relation authority blocked the unauthorized relation before any database interaction.
88.52%, BASE + counterfactual108 / 122
Answerable runtime TSAAnswerable queries that survived the real runtime and satisfied the result contract on BASE and every required counterfactual state.
100%, ambiguity 12/16 = 75.00%12 / 12
Policy blocksPolicy-blocked cases were refused exactly; ambiguity recognition is the weaker governance axis. Refusing SQL is a measured outcome, not a failure.

A demo answers syntax; this answers whether the query may run

A model can produce SQL that is syntactically fine and still unauthorized, unbounded or semantically wrong. DecisionSQL keeps the model’s proposal and the authority to execute separate, and evaluates governance apart from query execution because some requests should refuse SQL entirely.

Correctness is decided by running the query against changing database states, not by comparing it to a reference string. The benchmark measures first-pass behavior under a one-shot contract, so model decisions and server enforcement both stay visible.

What the system does

DecisionSQL covers the path from a governed request to a bounded result: one typed decision, a deterministic admission chain, and execution-based scoring across counterfactual states.

Governed decision

  • One typed decision per request: answer, clarification or block
  • Model-visible governed context, evaluator-only truth kept separate
  • Authority and policy blocks that never reach the SQL runtime

Deterministic SQL admission

  • sqlglot parse, SQL / object / function policy
  • Server-owned grain safety with a narrow deterministic normalizer
  • PostgreSQL EXPLAIN, a frozen cost gate and an accepted QueryPlan

Execution-based evaluation

  • Two reference witnesses and a typed ResultContract
  • Counterfactual fixtures that make wrong semantics diverge
  • Restricted read-only execution under reader role and timeout

One submission, a deterministic admission chain

Only an ANSWER + SQL submission enters the SQL runtime. The selected SQL then crosses each stage below before a restricted reader runs it, and the executor never accepts SQL directly from the model, the normalizer or the evaluator.

  1. 01

    One typed decision

    The model receives the question plus a governed context and emits exactly one decision. Runtime routing follows the parsed submission, not evaluator truth, so only ANSWER + SQL enters the SQL runtime.

  2. 02

    Parse and global policy

    sqlglot parses the selected SQL and the global policy enforces one read-only statement, governed object access, function restrictions and complexity limits before anything else runs.

  3. 03

    Request-scoped relation authority

    Global policy answers whether an object is queryable by the service; request-scoped authority answers whether this request may use it. An immutable relation-level ExecutionAuthority, derived from the same governed SchemaContext, rejects an unauthorized relation before any database connection, EXPLAIN or execution.

  4. 04

    Grain safety

    A GrainSafetyValidator detects parent-measure fanout, where a join to several child rows can silently multiply a SUM. Server-owned metadata, not the model, owns that contract.

  5. 05

    Narrow normalization, re-validated

    For the supported additive-parent, declared 1:N shape a deterministic normalizer preaggregates on the child side. Normalized SQL is never trusted automatically; it re-passes parse, policy and post-grain validation with no raw unsafe fallback.

  6. 06

    Cost admission

    PostgreSQL EXPLAIN runs on a deterministically analyzed state, and a frozen cost gate rejects SQL above the benchmark's max rows or max cost before execution.

  7. 07

    Accepted QueryPlan, then execution

    Execution requires an accepted immutable QueryPlan issued by the SQL safety service. A restricted reader runs it under a read-only transaction, reader role, statement timeout and bounded result rows.

Proposal and execution authority stay separate

The model owns the first-pass decision and, when it answers, the proposed SQL. It does not own authorization, physical schema truth, grain semantics or the right to execute. Deterministic software owns that half.

The model ownsDeterministic software owns
  • The first-pass typed decision
  • The proposed SQL when it chooses ANSWER
  • Nothing about authorization or execution
  • No physical schema truth, relationship authority or grain semantics
  • No right to run the query it wrote
  • Typed decision validation
  • SQL parsing, global policy and request-scoped relation authority
  • Narrow grain-safe normalization
  • PostgreSQL EXPLAIN, cost admission and QueryPlan creation
  • Restricted read-only execution and result comparison

Runtime routing follows the parsed submission, not evaluator truth. An ANSWER + SQL runs even when the truth is AMBIGUOUS; the harness records that runtime outcome separately instead of skipping the branch.

Architecture

A governed context feeds one typed decision. Only ANSWER + SQL enters the admission chain — parse, policy, grain safety, narrow normalization, EXPLAIN, a cost gate and an accepted QueryPlan — before restricted read-only execution. A separate evaluator scores the outcome off the request path against counterfactual states.

A natural-language analytics request is answered against a model-visible, governed context, and the model emits exactly one typed decision. Only an ANSWER + SQL submission enters the SQL runtime; clarifications and authority or policy blocks never do. The selected SQL then crosses a deterministic admission chain — sqlglot parse, global SQL policy, request-scoped relation authority, grain-safety validation, an optional narrow normalizer, re-validation, PostgreSQL EXPLAIN, a cost gate and an accepted immutable QueryPlan — before a restricted read-only executor runs it. Correctness is scored off the request path by executing against BASE and counterfactual database states, with evaluator-only truth that never reaches the model.

Governed decision

Runtime routing is driven by the parsed model submission, not by evaluator truth; only ANSWER + SQL reaches the SQL runtime.

Analytics Request with governed context Model-Visible Governed Context. Model-Visible Governed Context one-shot generation One-Shot LLM Decision. One-Shot LLM Decision enters SQL runtime ANSWER + SQL. One-Shot LLM Decision governance outcome NEEDS_CLARIFICATION. One-Shot LLM Decision governance outcome BLOCKED_AUTHORITY. One-Shot LLM Decision governance outcome BLOCKED_POLICY.

  1. ClientAnalytics Requestnatural language
  2. ControlModel-Visible Governed Contextpublic / governed schema, authorized metadata
  3. AnalyzerOne-Shot LLM Decisionuntrusted, typed, no retry
  1. ServiceANSWER + SQLread-only SELECT
    OutputNEEDS_CLARIFICATIONno SQL runtime
    BoundaryBLOCKED_AUTHORITYno SQL runtime
    BoundaryBLOCKED_POLICYno SQL runtime

Runtime admission (ANSWER + SQL)

The executor never accepts SQL directly from the model, the normalizer or the evaluator; execution requires an accepted immutable QueryPlan.

Selected SQL parses sqlglot Parse. sqlglot Parse policy Global SQL / Object Policy. Global SQL / Object Policy authorizes Request-Scoped Relation Authority. Request-Scoped Relation Authority checks grain GrainSafetyValidator. GrainSafetyValidator normalizes Grain-Safe Normalizer. Grain-Safe Normalizer re-validates Re-Parse + Re-Policy + Post-Grain. Re-Parse + Re-Policy + Post-Grain plans PostgreSQL EXPLAIN. PostgreSQL EXPLAIN cost gate Query Cost Gate. Query Cost Gate accepts Accepted QueryPlan. Accepted QueryPlan executes Restricted Read-Only Executor. Restricted Read-Only Executor returns Bounded Result.

  1. ServiceSelected SQLfrom the ANSWER submission
  2. Controlsqlglot Parseone statement, read-only
  3. ControlGlobal SQL / Object Policyis this object queryable at all
  4. ControlRequest-Scoped Relation Authoritymay this request use it, rejects before DB connection
  1. AnalyzerGrainSafetyValidatorparent-measure fanout detection
  2. ControlGrain-Safe Normalizernarrow supported shape, fail-closed
  3. ControlRe-Parse + Re-Policy + Post-Grainno raw unsafe fallback
  4. AnalyzerPostgreSQL EXPLAINplanner cost on analyzed state
  1. ControlQuery Cost Gatefrozen max rows / max cost
  2. PersistenceAccepted QueryPlanimmutable, issued by the SQL safety service
  3. ServiceRestricted Read-Only Executorreader role, statement timeout, bounded rows
  4. OutputBounded Resultrow-capped

Execution-based evaluation

Truth decides whether the governed decision is correct and whether result-contract evaluation applies; it does not decide whether SQL enters runtime.

Runtime + Governance Outcome scored against Execution-Based Evaluator. Execution-Based Evaluator uses Witnesses + Counterfactual Fixtures. Witnesses + Counterfactual Fixtures yields Governed Task Success.

  1. OutputRuntime + Governance Outcomerecorded per case
  2. AnalyzerExecution-Based Evaluatorevaluator-only truth, off the request path
  3. PersistenceWitnesses + Counterfactual Fixtures2 reference witnesses, typed ResultContract
  4. OutputGoverned Task Success160 / 180 on the benchmark

Architecture notes

  • Runtime routing is submission-driven: an ANSWER + SQL enters the SQL runtime even when the truth is AMBIGUOUS, so a wrong governed decision still produces a typed outcome instead of a crash.
  • The grain-safe normalizer is intentionally narrow — additive parent measure, a declared 1:N relationship and a supported LEFT JOIN fanout shape — and stays fail-closed or non-target outside it.
  • PostgreSQL ANALYZE runs as environment preparation before reader planning, outside SqlSafetyService, the cost gate, the executor and the request path, so EXPLAIN costs are deterministic and excluded from request latency.
  • Reference SQL, fixtures and expected results are evaluator-only and never enter the model request; correctness is execution-based semantic correctness, not SQL string or AST equality.

Four reliability decisions

Submission-driven runtime, truth-driven evaluation

Runtime routing is decided by the parsed model submission, not by evaluator truth. An ANSWER + SQL receives the full parse, policy, semantic, cost, QueryPlan and execution treatment even when the truth is AMBIGUOUS, and the harness records the runtime outcome separately instead of crashing on a wrong governed decision.

Server-owned grain safety

Joining a parent to several child rows can triple a parent measure. The model is not trusted to own that contract. Server-owned metadata describes entity and grain keys, relationship cardinality and aggregation behavior, and the normalizer stays fail-closed outside its frozen additive-parent, declared 1:N shape.

Execution-based correctness, not string match

Candidate SQL is not judged by exact text or AST equality. Two independent reference witnesses, a typed ResultContract and counterfactual fixtures that change distributions decide correctness, so SQL that accidentally returns the right rows on one state fails when the semantics actually diverge.

One-shot, with provenance discipline

One benchmark case yields one semantic attempt with no retry, repair, judge, selector or pass@K, which keeps model decision errors and server enforcement observable instead of letting a repair loop smooth them over. Responses are admitted under exact provider-request fingerprinting and deterministic replay.

Server-owned grain safety, made concrete

Joining one parent row to several child rows can silently multiply a parent measure. The model is not trusted to own this contract; server-owned metadata and a narrow deterministic normalizer are.

Naive join

Fanout multiplies the measure

order #A1amount 100
line 1× 100
line 2× 100
line 3× 100
SUM(order.amount) = 300counted 3×
Grain-safe

Child-side preaggregation

lines → subtotalpreaggregated
order #A1joined once
SUM(order.amount) = 100counted once

Additive parent measure, a declared 1:N relationship and a supported LEFT JOIN fanout shape resolve to deterministic child-side preaggregation. Outside that shape the normalizer stays fail-closed, and normalized SQL still re-passes parse, policy, EXPLAIN and the cost gate before it can run.

Server-owned grain contractNarrow supported shape onlyFail-closed outside itNo raw unsafe fallback

Governance is scored apart from execution

The benchmark includes cases where producing SQL is the wrong behavior, so refusing is a measured outcome, not a failure. The four behaviors are scored separately rather than merged into one accuracy number.

BehaviorCasesExpected decisionResult
ANSWERABLE122ANSWER + one read-only SELECT108 / 122 runtime TSA correct
AUTHORITY_BLOCKED30BLOCKED_AUTHORITY28 / 30, runtime blocks the unauthorized relation
AMBIGUOUS16NEEDS_CLARIFICATION12 / 16
POLICY_BLOCKED12BLOCKED_POLICY12 / 12

The three governance residuals are all in the ambiguity category; authority held at 15 / 15 with no unauthorized answers.

Selected benchmark evidence

Governed decisions, runtime admission, grain safety and semantic discrimination describe different failure surfaces. They stay separate instead of collapsing into one project score.

Evidence sliceCurrent resultWhat it covers
Governed task success160 / 180 = 88.89%One-shot governed decisions across four behavior classes on one 180-case benchmark spanning 12 synthetic domains. 20 governed misses overall.
Answerable runtime TSA108 / 122 = 88.52%Answerable queries that survived the real runtime and satisfied the result contract on BASE and every required counterfactual state. healthcare_10 passes BASE but fails a counterfactual.
Authority28 / 30 = 93.33%Model authority decisioning. telecom_15 is the known case where the model wrongly chose ANSWER; request-scoped relation authority rejected the unauthorized relation with zero database connection, EXPLAIN or execution.
Governance blockspolicy 12 / 12, ambiguity 12 / 16Policy blocking is exact at 100%; ambiguity recognition is the weaker axis at 75.00%. Refusing SQL is a measured, expected outcome on these cases.
Execution boundaryQueryPlan-gated, no raw unsafe fallbackAccepted execution requires an immutable QueryPlan issued by the SQL safety service; raw SQL or copied plan objects cannot bypass planning, and the reader runs under a read-only transaction, reader role, statement timeout and bounded rows.

Current implementation

RuntimePython 3.12 + FastAPI + PostgreSQL
SQL safetysqlglot parse + global policy + request authority + grain safety + cost gate
ExecutionAccepted QueryPlan + restricted read-only reader
DataSQLAlchemy + Alembic + synthetic enterprise packs
EvaluationCounterfactual fixtures + reference witnesses + result contracts
Qualitypytest + Ruff + mypy + OpenTelemetry

In short

What is DecisionSQL?
DecisionSQL is a governed one-shot Text-to-SQL system and execution-based benchmark where the model proposes a single typed decision — an answer with read-only SQL, a clarification, or an authority/policy block — and deterministic software owns parsing, policy, request-scoped relation authority, grain safety, cost admission and restricted execution before any SQL reaches PostgreSQL.
What problem does DecisionSQL solve?
A natural-language-to-SQL demo answers syntax. DecisionSQL is about whether a proposed query is authorized, bounded, semantically correct and safe to run: it keeps model proposal and execution authority separate, evaluates governance apart from query execution because some cases should refuse SQL entirely, and scores correctness by execution against counterfactual states rather than string similarity.

Deep dive

Repository README and walkthroughBenchmark specificationCurrent benchmark evaluationRuntime authority safety reportMachine-readable evaluation manifest
Open the full repository