Skip to content

dbt -> dpone inline self-service publishing

Overview

A dbt model can declare a governed delivery intent under the official dbt meta config. dbt remains the only editable source of truth. dpone reads the already-resolved value from target/manifest.json; it never parses SQL or implements a second config precedence engine.

flowchart LR
    MODEL["dbt SQL + schema.yml"] --> PARSE["dbt parse"]
    PARSE --> MANIFEST["manifest.json v10-v12"]
    MANIFEST --> CHECK["dpone dbt check"]
    CHECK --> COMPILE["dpone dbt compile"]
    COMPILE --> PACKS["immutable manifests + packs + dag-spec"]
    PACKS --> CACHE["fingerprinted Airflow cache"]
    CACHE --> DAG["dbt build gate -> parallel dpone transfers"]

Golden path

{{ config(
    materialized='table',
    contract={'enforced': true},
    meta={
        'dpone': {
            'publish': {
                'enabled': true,
                'profile': 'mssql_to_clickhouse_mart',
                'workflow': 'competitive_pricing'
            }
        }
    }
) }}

The platform profile supplies trusted connection refs, target defaults, runtime image, ClickHouse topology, DQ, lineage, and worker policy. Schedule, owner, timezone, and tags come from the workflow profile. Secrets, Vault paths, connection URIs, and Kubernetes Secret names are blocked in model metadata.

Commands

dbt parse
dpone dbt check --manifest target/manifest.json
dpone dbt explain --manifest target/manifest.json --model competitive_pricing
dpone dbt compile --manifest target/manifest.json --output-dir .dpone/gitops/airflow

Use --profiles when the registry is not at dpone/dbt-publish-profiles.yml. CI can also set DPONE_DBT_PUBLISH_PROFILES.

Build algorithm

  1. Validate dbt manifest schema v10-v12.
  2. Select only models with meta.dpone.publish.enabled: true.
  3. Validate the v1 intent, secret policy, materialization, and dbt contract.
  4. Resolve trusted publish/workflow profiles.
  5. Freeze strategy: auto into a concrete decision: incremental + unique_key -> incremental_merge, certified partition window -> partition_replace, otherwise full_refresh.
  6. Resolve ClickHouse engine/order/partition design. More than one replica defaults to ReplicatedMergeTree; explicit plain MergeTree is warned.
  7. Group models by workflow and reject conflicting workflow-level policy.
  8. Write one dbt build/test pack, one dpone transfer pack per model, and one DAG spec with static waves bounded by max_parallelism.
  9. Fingerprint artifacts for the existing remote release/index flow.

Runtime does not silently reconsider the selected strategy.

Failure and rollback

Failure Result
Unknown profile or invalid meta CI blocker, no artifacts published
dbt contract missing in production CI blocker
dbt build/test failure Transfer tasks do not start
Source schema drift dpone preflight fails before target mutation
Transfer or DQ failure Governed staging is aborted; final target remains unchanged
One workflow blocked Other successfully compiled workflows remain independent

Demo and macro

The macro only returns a canonical dictionary. It performs no I/O and is not required; direct meta remains the public contract.