ÖFK
Menu
All projects

Generative AI / RAG Platform

Working reference system

Knowledge Base RAG

A local-first multilingual RAG platform with tenant-scoped retrieval, measured reranking, strict answer validation, versioned index operations and a console that keeps the evidence path visible.

Retrieval evidence is useful only when access, provenance and release policy remain explicit.
View repository Local-first reference implementation with documented deployment limits
Canonical TechQA BGE-ON record70% USEFUL-ANSWER
End-to-end evaluationCorrect plus Partial, which is not an accuracy claim. 2% were materially incorrect and 28% released no useful supported answer.
Corrected TechQA evaluation0 CONTRACT FAILURES
Support-ID and citation contractsNo unknown, cross-query, hidden or unauthorized support IDs were accepted. Deterministic contract results on the evaluated corpus, not proven security.
220 multilingual questionsRECALL@5 1.0000
Reranker selection benchmarkAgainst 0.9563 cross-lingual without a reranker. A retrieval and ranking metric, separate from final-answer accuracy.
BGE_REMOVAL_NOT_SUPPORTEDREMOVAL NOT AUTHORIZED
Preregistered decision rule heldDisabling the reranker improved evidence completeness, but the semantic non-regression gate frozen beforehand failed, so the change was not adopted.

The answer is the end of a controlled evidence path

A relevant chunk is not automatically authorized, trustworthy or safe to release. The server resolves identity and tenant scope first, retrieval stays inside that boundary, and generated text is checked before the production path returns it.

The console exposes those decisions beside the answer. It is an operating and debugging surface for RAG, not a chat transcript with hidden infrastructure.

What the platform does

Knowledge Base RAG covers the path from source synchronization to an authorized, cited response, with the index and evaluation state available to operators.

Knowledge operations

  • Filesystem and Notion connectors
  • Incremental sync with content fingerprints
  • Versioned Qdrant collections and alias activation
  • Sync history, drift detection and rollback-aware cleanup

Retrieval and answers

  • Qwen3 dense embeddings plus BM25 sparse search
  • Native reciprocal rank fusion
  • Multilingual BGE reranking from 20 candidates to 5
  • Canonical citations and strict output validation

Control and evidence

  • Server-owned tenant ACL and role boundaries
  • Untrusted context serialization with answer_v3
  • Artifact-backed evaluation and security results
  • OpenTelemetry traces inspected through Jaeger

One query, eight inspectable stages

The production path keeps authorization, relevance and answer validation separate. This matters most when an authorized document contains instructions that the model should read as data, not follow as policy.

  1. 01

    Authenticated request

    FastAPI resolves the bearer token into a server-owned user, role and tenant context.

  2. 02

    Mandatory ACL

    The tenant filter is applied before reranking or generation. Later stages cannot widen the authorized set.

  3. 03

    Dense and sparse retrieval

    Qwen3 embeddings and Qdrant BM25 search produce separate candidate lists from the active index alias.

  4. 04

    RRF fusion

    Reciprocal rank fusion combines semantic and lexical signals without turning either score into a trust decision.

  5. 05

    Multilingual reranking

    BAAI/bge-reranker-v2-m3 reranks 20 authorized candidates and passes the best 5 onward.

  6. 06

    SectionAware evidence

    Surviving results are packed into request-scoped support units under a bounded context budget. Document text and metadata stay reference data and never receive a system or assistant role.

  7. 07

    Support-ID validation

    The model returns text plus support IDs. Unknown, cross-query, hidden and unauthorized IDs are rejected before anything else runs.

  8. 08

    Occurrence-aware validation

    Architecture V2 checks critical-value consistency against an immutable occurrence ledger, so a role decision stays attached to the occurrence it came from.

  9. 09

    Inspectable response

    The console keeps the answer, authorized sources, retrieval stages, security state and trace waterfall together.

Server controls stay outside retrieved content

Access control answers which chunks a user may retrieve. Prompt trust answers what the model may do with an authorized chunk. The system treats them as different boundaries and records both in the response projection.

Server-owned controlsUntrusted or presentational inputs
  • Bearer-token identity and role checks
  • Tenant and retrieval context
  • ACL filtering before reranking
  • Active index alias and pipeline fingerprint
  • Validation mode and release policy
  • Canonical citation membership checks
  • Document text, titles and headings
  • Source names and location metadata
  • Delimiter-looking instructions inside documents
  • Generated answer text before validation
  • Frontend identity selection and UI state
  • Client-observed timing data

Strict mode uses buffer, validate, release. Fast mode streams first and checks later, so it is a server-side opt-in for development and latency experiments, not a hidden frontend switch.

Architecture

The query path starts with server-owned identity and tenant scope, then moves through dense and sparse retrieval, RRF fusion, multilingual reranking, untrusted context construction and strict validation. A separate sync path builds compatible index versions before switching the active alias.

Filesystem and Notion sources enter a fingerprinted sync path that builds versioned Qdrant collections before the kb_active alias moves. Authenticated queries receive a server-owned tenant ACL before dense and sparse retrieval, RRF fusion and reranking, so no later relevance stage can widen the authorized set. SectionAware evidence construction turns the surviving results into request-scoped support units, and release is gated in three separate steps: support-ID validation, occurrence-aware critical-value validation, then citation resolution. The React console exposes the same retrieval, security, evaluation and trace state through read-only FastAPI projections.

Versioned index lifecycle

Source changes build and validate a compatible collection before the active alias moves.

Filesystem and Notion then connects to Connectors. Connectors source documents Parse · Chunk · Fingerprint. Parse · Chunk · Fingerprint compatible build Qwen3 + BM25. Qwen3 + BM25 writes inactive version Versioned Collection. Versioned Collection validates Compatibility Check. Compatibility Check activates or rolls back kb_active Alias.

  1. BoundaryFilesystemPDF · Markdown
    BoundaryNotionAPI connector
  2. ServiceConnectorsshared async protocol
  3. ControlParse · Chunk · Fingerprintcontent + pipeline identity
  1. ServiceQwen3 + BM25dense 1024 · sparse
  2. PersistenceVersioned CollectionQdrant · not yet active
  1. ControlCompatibility Checkschema · model · dimension
  2. Persistencekb_active Aliascurrent searchable index

Authorized retrieval and strict release

Identity and tenant scope constrain the candidate set before relevance scoring or generation begins.

React Console then connects to FastAPI Identity. FastAPI Identity creates retrieval context Mandatory Tenant ACL. Mandatory Tenant ACL filters active index Qdrant Retrieval. Qdrant Retrieval fuses RRF Fusion. RRF Fusion passes 20 BGE Reranker. BGE Reranker packs evidence SectionAware Support Units. SectionAware Support Units generates under answer_v3 Ollama Generation. Ollama Generation validates before release Support ID · Occurrence Validator · Citations. Support ID · Occurrence Validator · Citations releases Response Projection.

  1. ClientReact Consolequery + evidence inspector
  2. ControlFastAPI Identityuser · role · tenant
  3. BoundaryMandatory Tenant ACLbefore reranking
  1. PersistenceQdrant Retrievaldense + BM25 sparse
  2. AnalyzerRRF Fusionauthorized candidates
  1. AnalyzerBGE Rerankerauthorized top 20 → top 5
  2. BoundarySectionAware Support Unitsuntrusted reference data · request-scoped IDs
  3. ServiceOllama Generationbuffered in strict mode
  1. ControlSupport ID · Occurrence Validator · CitationsArchitecture V2 · release gate
  2. OutputResponse Projectionanswer · sources · security

Operations and evidence

Runtime traces and committed benchmark artifacts meet in read-only console projections.

Sync Runs and Query Traces and Evaluation Artifacts projects FastAPI UI Aggregation. FastAPI UI Aggregation renders Operations Console.

  1. ServiceSync Runshistory + lifecycle
    ServiceQuery Tracesrequest waterfall
    PersistenceEvaluation Artifactsmachine-readable results
  2. ObservabilityFastAPI UI Aggregationread-only projections
  3. ClientOperations Consolehealth · evidence · traces

Architecture notes

  • The reranker receives only ACL-filtered candidates and cannot widen the authorized set.
  • Strict production mode buffers, validates and then releases an answer; fast streaming is an explicit server-side alternative.
  • Pipeline fingerprints bind model, dimension, parser, index and chunk configuration to a compatible collection.
  • Citation integrity verifies membership in the authorized retrieved set, not claim-level semantic support.

Four reliability decisions

Authorization runs before relevance scoring

Tenant ACL filtering happens before the reranker sees a candidate. A strong relevance score cannot recover a chunk that the authenticated tenant was not allowed to retrieve.

Production answers are validated before release

Strict mode buffers generation until canonical citation and output-policy checks pass. Fast streaming remains an explicit server-side development option with a documented risk: output can reach the client before the post-stream check finishes.

The active index is a versioned artifact

The kb_active alias points to a compatible collection whose fingerprint covers embedding, parser, index and chunk settings. New collections are built and checked before activation, so a model or dimension change cannot silently reuse an incompatible index.

Pipeline changes need comparative evidence

The multilingual reranker was adopted after a paired 220-query benchmark. Token-aware chunking was implemented but the 500/50 baseline stayed in production because the current short corpus showed no measurable quality or efficiency gain.

Product proof from the operations console

These captures come from the public repository. Each one links to its source file so the interface and the evidence shown inside it can be inspected at full size.

RAG operations overview with document, chunk and source counts, recent synchronization runs, system health and active index details.
Overview: source health, recent syncs, security state and the active index are visible without entering the chat path.Open full-size image on GitHub Source: docs/assets/rag-overview.jpg
RAG playground showing a cited answer beside the exact retrieved source cards and stage timing.
Evidence inspector: the answer stays beside the ranked source set and the latency breakdown that produced it.Open full-size image on GitHub Source: docs/assets/rag-playground.jpg
RAG playground security panel showing the applied tenant ACL, isolated untrusted context and validate-before-release policy.
Security inspector: tenant authorization, context isolation and release policy are shown as separate controls.Open full-size image on GitHub Source: docs/assets/rag-playground-security.jpg
RAG playground trace inspector showing retrieval, reranking, generation and validation spans as a waterfall.
Trace waterfall: retrieval, reranking, generation and validation remain one inspectable request path.Open full-size image on GitHub Source: docs/assets/rag-playground-trace.jpg
RAG operations evaluation page showing the production baseline and the measured multilingual reranker decision.
Evaluation surface: active settings are tied back to committed benchmark artifacts and explicit adopt or keep decisions.Open full-size image on GitHub Source: docs/assets/rag-evaluations.jpg

The reranker changed because the measurement changed

A paired 220-query multilingual set compared the production choices on the same retrieval task. The selected BGE model improved cross-lingual recall and MRR, but its local CPU latency is much higher. The console shows both sides of that decision.

ConfigurationCross Recall@5Cross MRRTotal p95Decision
Reranking off0.95630.7448268.4 msBaseline
Previous cross-encoder0.45110.3670453.0 msReject
BGE multilingual1.00000.95582457.7 msAdopt

The selected configuration recorded 63 cross-lingual rescues and 0 drops in this benchmark. It is evidence for this dataset and runtime, not a universal ranking of rerankers.

Selected repository evidence

Test counts, retrieval quality, prompt security and generation sanity describe different failure surfaces. They remain separate instead of being compressed into one project score.

Evidence sliceCurrent resultWhat it covers
End-to-end answers70% useful · 2% incorrectCanonical TechQA BGE-ON record. Useful is Correct plus Partial, which is not an accuracy claim; strict full completeness scored separately at 30%.
Unavailable or abstained28% · 18% self · 10% forcedIntentional fail-closed behaviour when support cannot be established: 9/50 model self-abstentions and 5/50 deterministic forced abstentions. An abstention is not automatically a safety success or a quality failure; whether one was appropriate is a separate case-level question.
Support-ID and citation contracts0 failuresNo unknown, cross-query, hidden or unauthorized support IDs accepted. Deterministic contract results on the evaluated corpus, not proven security.
Candidate evidence recall95.9% at Top-20Measured at the shared candidate stage before reranking narrows to five, so it is not a statement about evidence in the model's final context.
Multilingual reranker220 queries · Recall@5 1.0000 · MRR 0.955863 cross-lingual rescues and 0 drops for the selected BGE model. A retrieval and ranking metric, separate from final-answer accuracy.
Preregistered gateBGE_REMOVAL_NOT_SUPPORTEDRemoving the reranker improved evidence completeness, but the semantic non-regression gate frozen beforehand failed, so the change was not adopted.
Repository verification844 backend · 18 frontendLast recorded full run; 2 external provider checks skipped, with Ruff, typecheck, lint and production build green.

From a retrieval pipeline to an operated knowledge system

Production RAG Platform established the original PDF retrieval, hybrid search, reranking, citations, evaluation and tracing path. This project kept that foundation and expanded the system boundary.

Earlier boundaryOne corpus and a focused retrieval path

Single-source ingestion, no tenant ACL and no operated index lifecycle.

Current boundaryTenant-scoped knowledge operations

Multi-source sync, versioned activation, security controls, benchmark artifacts and an operator console.

Current implementation

Operations consoleReact · TypeScript · Vite
APIFastAPI · HTTP · SSE · read-only UI aggregation
IdentityBearer tokens · tenant ACL · USER / OPERATOR / ADMIN roles
RetrievalQdrant dense + BM25 sparse · RRF · BGE multilingual reranker
ModelsQwen3-Embedding-4B @ 1024 · Ollama generation
Generationanswer_v3 · untrusted context · strict output policy
Index lifecycleSQLite registry · versioned collections · kb_active alias
ObservabilityOpenTelemetry · Jaeger · artifact-backed evaluations
RuntimePython 3.11+ · Docker Compose · native Ollama

Working system with explicit limits

The repository is a local-first engineering reference. Its measurements belong to the committed fixtures, models, hardware path and validation configuration.

Limits documented
  • The BGE reranker is a synchronous local model call inside the async retrieval path; concurrent model serving is not implemented.
  • The current chunking corpus is too short to distinguish the tested 256 to 768 token boundaries, so the 500/50 baseline remains active.
  • Citation integrity checks source membership, not claim-level semantic support. Calibrated answerability and abstention are also still open.
  • Whether an individual abstention was appropriate is assessed through layer-wise failure attribution, not inferred from the aggregate. Candidate evidence recall sits at an earlier retrieval boundary and does not classify individual abstentions.
  • Local tokens and development identities are demo authentication. A production deployment needs an external identity provider or verifier.
  • Sync coordination is process-local, and each configured source type maps to a server-owned tenant in the current connector model.

In short

What is Knowledge Base RAG?
Knowledge Base RAG is a local-first platform for operating multilingual knowledge bases. Server-owned tenant ACL runs before retrieval and reranking, SectionAware evidence construction turns authorized results into request-scoped support units, and an occurrence-aware validator checks critical-value consistency before a citation-resolved answer is released. Retrieval, evidence, generation, validation and citation failures are attributed to distinct boundaries rather than collapsed into one pass or fail label.
What problem does Knowledge Base RAG solve?
Retrieval quality alone does not make an answer trustworthy. The system has to preserve tenant boundaries, construct evidence deliberately, explain which support units reached the model, validate the model's references, and fail closed when a safety boundary cannot be established. The second half of the project is treating evaluation as an engineering control rather than a scoreboard: decision rules are frozen before results are known, rejected candidates are preserved as evidence instead of being rewritten into a success narrative, and metric families are kept separate so a ranking score is never read as answer accuracy.

Deep dive

ArchitectureSecurity modelReranking decisionChunking decisionEmbedding migrationArchitecture decisions
Open the full repository
Related writingBuilding Citation Integrity into a Production RAG PipelineHow source identity, bounded context and citation checks turn a reference into testable system behavior.RAG Can Provide Evidence. It Cannot Grant Authority.Why retrieved text can support an answer without becoming a permission or policy channel.