dbt integration¶

dpone integrates with dbt through dbt artifacts. It does not embed or replace dbt. The recommended production pattern is:

Run dpone source/sink loads.
Run dbt build for transformations/tests.
Export dbt lineage evidence from target/manifest.json and target/run_results.json.
Publish dpone and dbt OpenLineage events to the same collector/catalog.

This keeps runtime boundaries clean: dpone owns extract/load/state/evidence, while dbt owns SQL transformations, refs, sources, tests, snapshots, and model selection.

Quickstart¶

dbt build --project-dir analytics --profiles-dir analytics

dpone ops dbt-lineage \
  --output-dir .dpone/dbt-lineage/orders \
  --manifest analytics/target/manifest.json \
  --run-results analytics/target/run_results.json \
  --run-registry-entry .dpone/run-registry/<run_id>__run_registry.json \
  --namespace dbt.local \
  --format json

Artifacts written:

dbt_lineage.json
dbt_lineage.md
dbt_openlineage.json

Lineage model¶

flowchart LR
    DponeRun["dpone run"] --> Registry["dpone run registry"]
    DbtBuild["dbt build"] --> Manifest["target/manifest.json"]
    DbtBuild --> Results["target/run_results.json"]
    Registry --> Export["dpone ops dbt-lineage"]
    Manifest --> Export
    Results --> Export
    Export --> Graph["dbt_lineage.json"]
    Export --> Events["dbt_openlineage.json"]
    Events --> Collector["OpenLineage collector / catalog"]

dbt_lineage.json contains:

Field	Meaning
`nodes`	dbt sources, models, seeds, snapshots, and tests.
`edges`	`depends_on.nodes` relationships from upstream to downstream.
`run_id`	dpone run id when `--run-registry-entry` is provided; otherwise dbt invocation id.
`invocation_id`	dbt invocation id from `run_results.json`.

dbt_openlineage.json contains one event per executable dbt dataset node:

dbt resource type	OpenLineage behavior
`model`	Emits one event with upstream refs/sources as inputs and model relation as output.
`seed`	Emits one event with dependencies as inputs and seed relation as output.
`snapshot`	Emits one event with dependencies as inputs and snapshot relation as output.
`source`	Used as an input dataset, does not emit its own event.
`test`	Included in graph and status evidence, does not emit a dataset-producing event.

dbt artifact requirements¶

Artifact	Required	Purpose
`target/manifest.json`	yes	Graph structure, nodes, sources, refs, relation names.
`target/run_results.json`	recommended	dbt statuses, invocation id, failed model/test detection.
dpone run registry entry	recommended	Joins dbt lineage to dpone `run_id`, evidence, and checksums.

Failure behavior¶

Condition	CLI status	Blocker
Missing `manifest.json`	red	`dbt_manifest.missing`
Invalid `manifest.json`	red	`dbt_manifest.invalid_json`
dbt model/test has failed status	red	`dbt_results.not_passed`
Attached dpone run registry is red	red	`run_registry.not_passed`
Attached run registry is missing	red	`run_registry.missing`

Example CI step¶

dpone run manifests/orders_load.yml \
  --selector orders \
  --format json > .dpone/runs/orders/run_result.json

dpone ops run-registry \
  --output-dir .dpone/run-registry \
  --run-result .dpone/runs/orders/run_result.json \
  --format json

dbt build --project-dir analytics --profiles-dir analytics

dpone ops dbt-lineage \
  --output-dir .dpone/dbt-lineage/orders \
  --manifest analytics/target/manifest.json \
  --run-results analytics/target/run_results.json \
  --run-registry-entry .dpone/run-registry/orders__run_registry.json \
  --namespace dbt.local \
  --format json

Python API¶

from dpone.ops.dbt_lineage import DbtLineageService

report = DbtLineageService().export(
    output_dir=".dpone/dbt-lineage/orders",
    manifest_path="analytics/target/manifest.json",
    run_results_path="analytics/target/run_results.json",
    run_registry_entry_path=".dpone/run-registry/run_01__run_registry.json",
    namespace="dbt.local",
)

if not report.passed:
    raise RuntimeError(report.blockers)

Runbook¶

If dbt_manifest.missing, run dbt build or check --project-dir.
If dbt_results.not_passed, fix failed dbt models/tests before publishing release evidence.
If relation names are empty, confirm dbt adapter generated relation_name in the manifest.
If catalog datasets do not join with dpone datasets, standardize namespace names across lineage-export and dbt-lineage.
Attach dbt_lineage.json, dbt_openlineage.json, and dbt_lineage.md to release evidence for production changes.