ÖFK
Menu
All projects

Generative AI / RAG Platform

Working reference system

Knowledge Base RAG

A local-first multilingual RAG platform with tenant-scoped retrieval, measured reranking, strict answer validation, versioned index operations and a console that keeps the evidence path visible.

Retrieval evidence is useful only when access, provenance and release policy remain explicit.
View repository Local-first reference implementation with documented deployment limits
2 external checks skipped844 BACKEND TESTS
Last recorded full verification18 frontend tests also passed, with Ruff, typecheck, lint and the production build green.
Reranker decision220 QUERIES
Paired multilingual evaluationThe selected BGE model recorded cross-lingual Recall@5 1.0000, MRR 0.9558, 63 rescues and 0 drops.
Committed security suite82 ADVERSARIAL CASES
Prompt-security evidenceInjection, spoofing, suppression, unauthorized citation and cross-tenant exfiltration rates all recorded 0.0000.
Baseline generation path26/26 SUCCESSFUL
Generation sanityCitation integrity, not-found behavior and strict validation each recorded 1.0000 in the exercised suite.

The answer is the end of a controlled evidence path

A relevant chunk is not automatically authorized, trustworthy or safe to release. The server resolves identity and tenant scope first, retrieval stays inside that boundary, and generated text is checked before the production path returns it.

The console exposes those decisions beside the answer. It is an operating and debugging surface for RAG, not a chat transcript with hidden infrastructure.

What the platform does

Knowledge Base RAG covers the path from source synchronization to an authorized, cited response, with the index and evaluation state available to operators.

Knowledge operations

  • Filesystem and Notion connectors
  • Incremental sync with content fingerprints
  • Versioned Qdrant collections and alias activation
  • Sync history, drift detection and rollback-aware cleanup

Retrieval and answers

  • Qwen3 dense embeddings plus BM25 sparse search
  • Native reciprocal rank fusion
  • Multilingual BGE reranking from 20 candidates to 5
  • Canonical citations and strict output validation

Control and evidence

  • Server-owned tenant ACL and role boundaries
  • Untrusted context serialization with answer_v3
  • Artifact-backed evaluation and security results
  • OpenTelemetry traces inspected through Jaeger

One query, eight inspectable stages

The production path keeps authorization, relevance and answer validation separate. This matters most when an authorized document contains instructions that the model should read as data, not follow as policy.

  1. 01

    Authenticated request

    FastAPI resolves the bearer token into a server-owned user, role and tenant context.

  2. 02

    Mandatory ACL

    The tenant filter is applied before reranking or generation. Later stages cannot widen the authorized set.

  3. 03

    Dense and sparse retrieval

    Qwen3 embeddings and Qdrant BM25 search produce separate candidate lists from the active index alias.

  4. 04

    RRF fusion

    Reciprocal rank fusion combines semantic and lexical signals without turning either score into a trust decision.

  5. 05

    Multilingual reranking

    BAAI/bge-reranker-v2-m3 reranks 20 authorized candidates and passes the best 5 onward.

  6. 06

    Untrusted context

    Document text and metadata are serialized as reference data. They never receive a system or assistant role.

  7. 07

    Strict validation

    The production path buffers the answer, validates citations and output policy, then releases it.

  8. 08

    Inspectable response

    The console keeps the answer, authorized sources, retrieval stages, security state and trace waterfall together.

Server controls stay outside retrieved content

Access control answers which chunks a user may retrieve. Prompt trust answers what the model may do with an authorized chunk. The system treats them as different boundaries and records both in the response projection.

Server-owned controlsUntrusted or presentational inputs
  • Bearer-token identity and role checks
  • Tenant and retrieval context
  • ACL filtering before reranking
  • Active index alias and pipeline fingerprint
  • Validation mode and release policy
  • Canonical citation membership checks
  • Document text, titles and headings
  • Source names and location metadata
  • Delimiter-looking instructions inside documents
  • Generated answer text before validation
  • Frontend identity selection and UI state
  • Client-observed timing data

Strict mode uses buffer, validate, release. Fast mode streams first and checks later, so it is a server-side opt-in for development and latency experiments, not a hidden frontend switch.

Architecture

The query path starts with server-owned identity and tenant scope, then moves through dense and sparse retrieval, RRF fusion, multilingual reranking, untrusted context construction and strict validation. A separate sync path builds compatible index versions before switching the active alias.

Filesystem and Notion sources enter a fingerprinted sync path that builds versioned Qdrant collections before the kb_active alias moves. Authenticated queries receive a server-owned tenant ACL before dense and sparse retrieval, RRF fusion, multilingual reranking, untrusted context construction and strict answer validation. The React console exposes the same retrieval, security, evaluation and trace state through read-only FastAPI projections.

Versioned index lifecycle

Source changes build and validate a compatible collection before the active alias moves.

Filesystem and Notion then connects to Connectors. Connectors source documents Parse · Chunk · Fingerprint. Parse · Chunk · Fingerprint compatible build Qwen3 + BM25. Qwen3 + BM25 writes inactive version Versioned Collection. Versioned Collection validates Compatibility Check. Compatibility Check activates or rolls back kb_active Alias.

  1. BoundaryFilesystemPDF · Markdown
    BoundaryNotionAPI connector
  2. ServiceConnectorsshared async protocol
  3. ControlParse · Chunk · Fingerprintcontent + pipeline identity
  1. ServiceQwen3 + BM25dense 1024 · sparse
  2. PersistenceVersioned CollectionQdrant · not yet active
  1. ControlCompatibility Checkschema · model · dimension
  2. Persistencekb_active Aliascurrent searchable index

Authorized retrieval and strict release

Identity and tenant scope constrain the candidate set before relevance scoring or generation begins.

React Console then connects to FastAPI Identity. FastAPI Identity creates retrieval context Mandatory Tenant ACL. Mandatory Tenant ACL filters active index Qdrant Retrieval. Qdrant Retrieval fuses RRF Fusion. RRF Fusion passes 20 BGE Reranker. BGE Reranker serializes Untrusted Context. Untrusted Context generates under answer_v3 Ollama Generation. Ollama Generation validates before release Strict Validation. Strict Validation releases Response Projection.

  1. ClientReact Consolequery + evidence inspector
  2. ControlFastAPI Identityuser · role · tenant
  3. BoundaryMandatory Tenant ACLbefore reranking
  1. PersistenceQdrant Retrievaldense + BM25 sparse
  2. AnalyzerRRF Fusionauthorized candidates
  1. AnalyzerBGE Rerankermultilingual · top 5
  2. BoundaryUntrusted Contextreference data only
  3. ServiceOllama Generationbuffered in strict mode
  1. ControlStrict Validationcitations + output policy
  2. OutputResponse Projectionanswer · sources · security

Operations and evidence

Runtime traces and committed benchmark artifacts meet in read-only console projections.

Sync Runs and Query Traces and Evaluation Artifacts projects FastAPI UI Aggregation. FastAPI UI Aggregation renders Operations Console.

  1. ServiceSync Runshistory + lifecycle
    ServiceQuery Tracesrequest waterfall
    PersistenceEvaluation Artifactsmachine-readable results
  2. ObservabilityFastAPI UI Aggregationread-only projections
  3. ClientOperations Consolehealth · evidence · traces

Architecture notes

  • The reranker receives only ACL-filtered candidates and cannot widen the authorized set.
  • Strict production mode buffers, validates and then releases an answer; fast streaming is an explicit server-side alternative.
  • Pipeline fingerprints bind model, dimension, parser, index and chunk configuration to a compatible collection.
  • Citation integrity verifies membership in the authorized retrieved set, not claim-level semantic support.

Four reliability decisions

Authorization runs before relevance scoring

Tenant ACL filtering happens before the reranker sees a candidate. A strong relevance score cannot recover a chunk that the authenticated tenant was not allowed to retrieve.

Production answers are validated before release

Strict mode buffers generation until canonical citation and output-policy checks pass. Fast streaming remains an explicit server-side development option with a documented risk: output can reach the client before the post-stream check finishes.

The active index is a versioned artifact

The kb_active alias points to a compatible collection whose fingerprint covers embedding, parser, index and chunk settings. New collections are built and checked before activation, so a model or dimension change cannot silently reuse an incompatible index.

Pipeline changes need comparative evidence

The multilingual reranker was adopted after a paired 220-query benchmark. Token-aware chunking was implemented but the 500/50 baseline stayed in production because the current short corpus showed no measurable quality or efficiency gain.

Product proof from the operations console

These captures come from the public repository. Each one links to its source file so the interface and the evidence shown inside it can be inspected at full size.

RAG operations overview with document, chunk and source counts, recent synchronization runs, system health and active index details.
Overview: source health, recent syncs, security state and the active index are visible without entering the chat path.Open full-size image on GitHub Source: docs/assets/rag-overview.jpg
RAG playground showing a cited answer beside the exact retrieved source cards and stage timing.
Evidence inspector: the answer stays beside the ranked source set and the latency breakdown that produced it.Open full-size image on GitHub Source: docs/assets/rag-playground.jpg
RAG playground security panel showing the applied tenant ACL, isolated untrusted context and validate-before-release policy.
Security inspector: tenant authorization, context isolation and release policy are shown as separate controls.Open full-size image on GitHub Source: docs/assets/rag-playground-security.jpg
RAG playground trace inspector showing retrieval, reranking, generation and validation spans as a waterfall.
Trace waterfall: retrieval, reranking, generation and validation remain one inspectable request path.Open full-size image on GitHub Source: docs/assets/rag-playground-trace.jpg
RAG operations evaluation page showing the production baseline and the measured multilingual reranker decision.
Evaluation surface: active settings are tied back to committed benchmark artifacts and explicit adopt or keep decisions.Open full-size image on GitHub Source: docs/assets/rag-evaluations.jpg

The reranker changed because the measurement changed

A paired 220-query multilingual set compared the production choices on the same retrieval task. The selected BGE model improved cross-lingual recall and MRR, but its local CPU latency is much higher. The console shows both sides of that decision.

ConfigurationCross Recall@5Cross MRRTotal p95Decision
Reranking off0.95630.7448268.4 msBaseline
Previous cross-encoder0.45110.3670453.0 msReject
BGE multilingual1.00000.95582457.7 msAdopt

The selected configuration recorded 63 cross-lingual rescues and 0 drops in this benchmark. It is evidence for this dataset and runtime, not a universal ranking of rerankers.

Selected repository evidence

Test counts, retrieval quality, prompt security and generation sanity describe different failure surfaces. They remain separate instead of being compressed into one project score.

Evidence sliceCurrent resultWhat it covers
Repository verification844 backend · 18 frontendLast recorded full run; 2 external provider checks skipped, with Ruff, typecheck, lint and production build green.
Multilingual reranker220 queries · Recall@5 1.0000 · MRR 0.955863 cross-lingual rescues and 0 drops for the selected BGE model in the committed paired benchmark.
Prompt security82 adversarial casesInjection, spoofing, citation suppression, unauthorized citation and cross-tenant exfiltration rates were all 0.0000 in this suite.
Generation sanity26/26 successfulCitation integrity, not-found behavior and strict validation each recorded 1.0000 for the exercised baseline path.

From a retrieval pipeline to an operated knowledge system

Production RAG Platform established the original PDF retrieval, hybrid search, reranking, citations, evaluation and tracing path. This project kept that foundation and expanded the system boundary.

Earlier boundaryOne corpus and a focused retrieval path

Single-source ingestion, no tenant ACL and no operated index lifecycle.

Current boundaryTenant-scoped knowledge operations

Multi-source sync, versioned activation, security controls, benchmark artifacts and an operator console.

Current implementation

Operations consoleReact · TypeScript · Vite
APIFastAPI · HTTP · SSE · read-only UI aggregation
IdentityBearer tokens · tenant ACL · USER / OPERATOR / ADMIN roles
RetrievalQdrant dense + BM25 sparse · RRF · BGE multilingual reranker
ModelsQwen3-Embedding-4B @ 1024 · Ollama generation
Generationanswer_v3 · untrusted context · strict output policy
Index lifecycleSQLite registry · versioned collections · kb_active alias
ObservabilityOpenTelemetry · Jaeger · artifact-backed evaluations
RuntimePython 3.11+ · Docker Compose · native Ollama

Working system with explicit limits

The repository is a local-first engineering reference. Its measurements belong to the committed fixtures, models, hardware path and validation configuration.

Limits documented
  • The BGE reranker is a synchronous local model call inside the async retrieval path; concurrent model serving is not implemented.
  • The current chunking corpus is too short to distinguish the tested 256 to 768 token boundaries, so the 500/50 baseline remains active.
  • Citation integrity checks source membership, not claim-level semantic support. Calibrated answerability and abstention are also still open.
  • Local tokens and development identities are demo authentication. A production deployment needs an external identity provider or verifier.
  • Sync coordination is process-local, and each configured source type maps to a server-owned tenant in the current connector model.

Deep dive

ArchitectureSecurity modelReranking decisionChunking decisionEmbedding migrationArchitecture decisions
Open the full repository
Related writingBuilding Citation Integrity into a Production RAG PipelineHow source identity, bounded context and citation checks turn a reference into testable system behavior.RAG Can Provide Evidence. It Cannot Grant Authority.Why retrieved text can support an answer without becoming a permission or policy channel.