All projects

Data Engineering / Lineage

current

dbt Feature Lineage

Lineage is useful when it explains the blast radius of a change, not only when it draws a graph.

Local-first dbt analysis for model dependencies, cross-model column lineage, change impact and query flow without a live warehouse.

MANIFEST + STATIC2 MODES
One normalized project modelManifest and compiled SQL are preferred when available; direct SQL and YAML analysis keeps the path useful without generated artifacts.
NEXT.JS WEB APP5 VIEWS
One shared selection contextDashboard, Model Explorer, Model DAG, Column Lineage and Feature Explorer share the same project and model-group selection.
DEMO STATIC PATHLOCAL ONLY
No warehouse connection requiredThe demo project runs through static analysis, so the exploration loop stays local and the dbt project is not changed.

The problem

Make a dbt project explain itself.

A large dbt project can make a simple question expensive: where did this column come from, and what changes if I touch it? The answer is spread across model SQL, YAML metadata, joins, CTEs and indirect consumers.

This tool turns that repository structure into a local exploration loop. It reads the project, normalizes what it finds, then exposes the same representation to a CLI and an interactive UI.

Artifact-first, fallback-ready

Loaders prefer dbt's manifest and catalog artifacts, then fall back to recursive SQL/YAML scanning instead of making the whole experience depend on a successful dbt parse.

Impact is not one number

Downstream analysis separates models that reference a column directly from the full transitive chain, giving review decisions a useful blast-radius boundary.

One analysis layer, two interfaces

Typer and the FastAPI-backed Next.js web app consume shared domain models and services. The CLI and web app do not carry separate lineage logic that can drift apart.

Readable partial results

The parsing strategy returns partial results with a warning when a query cannot be fully understood, making uncertainty visible rather than silently dropping a model.

Input boundary

Two inputs. One domain model.

The tool prefers the artifacts dbt already understands, but does not make artifact generation a hard prerequisite. Both paths converge before graph construction and user-facing analysis.

01

Manifest mode

target/manifest.json + catalog.json

Uses dbt's own dependency and compiled-SQL artifacts when they exist. `--generate-artifacts` can run `dbt parse` on demand.

02

Static mode

SQL + YAML on disk

Discovers model layers, sources, ref()/source() dependencies, output columns and SQL structures without executing macros or requiring a warehouse.

In both modes, `ref()` and `source()` dependencies are extracted, Jinja relations are replaced with SQL-safe placeholders, and the resulting SQL is parsed with sqlglot.

Product surface

From project scan to feature comparison.

The Next.js web app keeps the project and optional model-group selection stable while each view answers a different inspection question.

01

Select Project

Point at a local directory or clone from a git URL, choose a dbt project and optionally scope the app to a model group.

02

Model Explorer

Inspect one model through overview, query flow, columns and raw SQL views.

03

Model DAG

See model-level dependencies and inspect materialization, owner, tests, description and tags.

04

Column Lineage

Trace a column upstream to raw sources or downstream to consumers, with an impact summary for the latter.

05

Feature Explorer

Compare every model producing a given column name by layer, description, owner, tags and test count.

A global ⌘K command palette provides the fast path across models, columns and pages. Each selection is represented by a shareable query-string URL rather than client-only state.

Impact semantics

Direct and transitive impact stay separate.

A column change does not affect every downstream node in the same way. The downstream report groups the chain by model and preserves the distinction between a consumer that references the column and a model that inherits the change later in the graph.

QuestionAnswer surfaceWhy it matters
Where did this column come from?Upstream lineageWalk joins, coalesces and renames back to raw sources.
Who consumes it directly?Direct impactPrioritize the models that reference the changed column themselves.
What inherits the change?Transitive chainSee the complete downstream reach without treating every node as an equal review target.
Where is the same feature produced?Feature ExplorerCompare model, layer, owner, description, tags and test count side by side.

Architecture

Load once. Analyze everywhere.

Loaders resolve the project into shared domain models. Services then build model DAGs, column lineage, traces, impact summaries, query-flow steps and model health for both the Typer CLI and the FastAPI-backed Next.js web app.

A local dbt project or cloned git repository is resolved into manifest-aware or static analysis inputs. Both modes normalize into shared domain models. A common service layer powers model DAGs, column lineage, tracing, impact analysis, query flow and model health for both the Typer CLI and the FastAPI-backed Next.js web app.

Two inputs, one domain model

Artifact-first loading and static fallback expose the same project representation to every downstream analysis.

dbt project / git clone load Manifest Mode and Static Mode. Manifest Mode and Static Mode normalize Shared Domain Models.

  1. Boundarydbt project / git clonelocal checkout boundary
  2. ServiceManifest Modetarget/manifest.json · compiled SQL
    ServiceStatic ModeSQL + YAML scanners / parsers
  3. PersistenceShared Domain Modelssame representation

Shared analysis services

CLI and the FastAPI-backed web app consume one service layer rather than reimplementing analysis in each interface.

Domain Models analyze Services. Services serve Typer CLI (human + JSON output). Services serve FastAPI → Next.js (JSON API + web app).

  1. PersistenceDomain Modelsmanifest or static
  2. AnalyzerServicesshared analysis layer
    • Dashboard + health
    • Model DAG
    • Column lineage
    • Impact + exposures
    • Query flow
  3. ClientTyper CLIhuman + JSON output
    ClientFastAPI → Next.jsJSON API + web app

Architecture notes

  • Manifest and static analysis normalize into the same domain models.
  • Typer and the FastAPI-backed Next.js app share one service layer; analysis logic is not duplicated between interfaces.
  • The demo and static-analysis path require no live data warehouse connection, and git imports are handled through the local clone boundary.

Design decisions

The system keeps uncertainty visible.

01

Normalize before analyzing

Manifest-aware and static loading resolve into the same domain representation. Every graph, trace and query-flow view can therefore stay ignorant of where the project came from.

02

Keep analysis outside the presentation layer

The service layer builds schema and lineage graphs, model DAGs, column search, query-flow steps, impact summaries and model health. Typer and the FastAPI backend remain thin adapters over that behavior, with Next.js handling presentation.

03

Treat parsing uncertainty as product state

The parser handles Jinja relations through SQL-safe placeholders and reports partial results when a query is not fully parseable. A warning is more useful than a confident blank screen.

Product proof

See the working surface, not just the architecture.

These screenshots come from the repository's current Next.js walkthrough. Each image links back to its original GitHub file so the full-size source remains available on desktop and mobile.

dbt Feature Lineage dashboard showing model, column and dependency counts, model health, layers, materializations and documentation coverage.
Dashboard: project scale, model health and documentation gaps are visible in one overview.Open full-size source on GitHub Source: docs/photos/web-dashboard.jpg
dbt Feature Lineage column lineage view showing a customer_id path from a staging model through transformations to a downstream mart.
Column lineage: source path, transformations and downstream impact stay together.Open full-size source on GitHub Source: docs/photos/web-column-lineage.jpg
dbt Feature Lineage model DAG showing staging, intermediate and marts models with a details panel.
Model DAG: graph context and selected-node detail share the same inspection surface.Open full-size source on GitHub Source: docs/photos/web-model-dag.jpg
dbt Feature Lineage project selector showing local directory and clone from git options, a selected project and manifest mode.
Select project: local directory and clone-from-git inputs converge on the same project context.Open full-size source on GitHub Source: docs/photos/web-select-project.jpg

Reproducible path

Short path from clone to inspection.

The repository documents a Docker-first development loop. Build the image, run the test suite, start the FastAPI backend, then run the Next.js frontend against the included sample project.

QUICK STARTFASTAPI + NEXT.JS
make build
make test
make api
make web
dbt-feature-lineage analyze examples/sample_banking_dbt
dbt-feature-lineage lineage examples/sample_banking_dbt customer_id --direction downstream --impact
Release historyv0.1 → v0.8 complete

Core input, lineage, impact, visualization and explorer milestones are marked complete in the repository roadmap.

Quality loopBuild · test · lint

Docker, pytest and Ruff are first-class Makefile commands in the documented development path.

Stack

Small surface, clear responsibilities.

BackendPython 3.12 · FastAPI · Typer · Uvicorn
Analysisdbt Core · sqlglot · NetworkX · Jinja2 · PyYAML · Pydantic
FrontendNext.js · TypeScript · Tailwind CSS · React Flow
DeliveryDocker · Docker Compose · Make
Qualitypytest · Ruff

Limits

Useful because the boundary is explicit.

This is a local exploration tool, not a replacement for dbt Cloud, observability or a transformation engine. The repository is direct about where analysis can diverge from a fully compiled project.

  • No dbt Cloud, Airflow or warehouse integration.
  • Private git imports rely on the host's configured credential helper or SSH agent; the app does not store tokens.
  • Complex custom macros may not parse correctly.
  • Static analysis can differ from compiled dbt SQL when no manifest is available.
  • Projects that depend on generated or unavailable files may produce incomplete analysis.

Deep dive

Follow the implementation.