Compose a native workspace and independent workloads

For data and platform engineers, release composition publishes one immutable release containing a complete native multi-project dbt workspace and a separate ordinary workload catalog. It verifies both source inventories before publication. Use this guide when the ordinary transfers are independently owned, rather than transfers generated by a dbt project's publishing policy.

Composition prepares delivery artifacts. It does not execute SQL, activate a deployment, or certify a live route. See the contract reference for supported inputs and operations guide for recovery, delivery, and upgrades.

Prepare the workspace and toolchain

Use compatible composition-aware core, Airflow provider, and runtime packages. Older readers reject release-set.v3. Keep the exact producer version recorded in the resulting release; changing the producer version can change the release ID.

Start in your repository root with a complete workspace configured through workspace authoring. That guide covers project publishing policies, the pinned dbt toolchain, and local manifest preparation. The compact workspace delivery guide explains the native input required here. Use distinct DAG IDs, workload IDs, artifact paths, and logical target relations across the workspace and ordinary workloads.

Set XCOM_SIDECAR_IMAGE to your approved digest-pinned sidecar reference before running the following commands. Native compilation and ordinary inventory do not pull images; actual runtime images and connection bindings require separate platform preparation.

mkdir -p reports
dpone dbt workspace compile \
  --root workspace --output-dir compiled-native --format json \
  > reports/native-compile.json

dpone gitops airflow release-materialize \
  --pack-root compiled-native --cache-root native-cache \
  --xcom-sidecar-image "$XCOM_SIDECAR_IMAGE" --format json \
  > reports/native-materialize.json

Check each command's exit status before continuing. The second report must have passed: true; retain its release_id and release_dir. This compact native release is the input to composition. Preserve its entire tree, including release-set.json, source snapshot, integrity subject, schemas, and runtime payloads. Do not copy selected dbt packs into a legacy _dags root.

Build an ordinary catalog through public producers

The following files describe one synthetic PostgreSQL transfer. source and target are logical connection references; no credentials are embedded. The image below is a synthetic digest-shaped reference for offline preparation; replace it with your platform's approved immutable runtime image before delivery.

Create ordinary/manifests/orders.yaml:

name: ordinary_orders
source:
  type: postgres
  connection_ref: source
  query: SELECT 1 AS id
sink:
  type: postgres
  connection_ref: target
  table:
    schema: public
    name: ordinary_orders
  strategy:
    mode: full_refresh

Create ordinary/gitops.yaml:

gitops:
  version: 1
  defaults:
    image: example.invalid/dpone@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
    image_digest: sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
  environments:
    dev:
      namespace: airflow-example
      runner_policy: advisory
  includes:
    - path: domains/*.yaml

Create ordinary/domains/sample.yaml:

domain: sample
workloads:
  ordinary_orders:
    manifest: ../manifests/orders.yaml
dags:
  ordinary_daily:
    schedule: null
    start_date: "2026-01-01"
    workloads: [ordinary_orders]

The ordinary composition capability admits a plain single-transfer manifest per workload, with optional referenced SQL files. It does not admit batch or authoring manifests, recipe expansion, hooks, transforms, custom runner assets, or dbt execution. Leave connection projection unconfigured: the final deployment owns runtime connection delivery. A secret-volume or Airflow environment bridge is not an interchangeable input to this procedure.

Inspect and compile the catalog:

dpone gitops workloads list --workload-set ordinary/gitops.yaml --env dev

dpone gitops airflow pack \
  --workload ordinary_orders --workload-set ordinary/gitops.yaml --env dev \
  --output-path .dpone/ordinary-preview/ordinary_orders/airflow-pack.json \
  --format json > reports/ordinary-pack.json

dpone gitops airflow reconcile \
  --workload-set ordinary/gitops.yaml --all-workloads --env dev \
  --output-dir .dpone/ordinary-build --format json \
  > reports/ordinary-reconcile.json

The single-pack command is an inspection step. Use the full reconcile output for composition. Its pack root is .dpone/ordinary-build/airflow, containing _dags/ordinary_daily.dag-spec.json and ordinary_orders/airflow-pack.json. Keep reports outside that root. Use a fresh reconcile destination when changing the catalog so stale workload directories do not become orphan artifacts. The catalog guide and reconcile runbook describe these producers.

Capture source identity and compose

dpone gitops airflow release-inventory \
  --pack-root .dpone/ordinary-build/airflow \
  --xcom-sidecar-image "$XCOM_SIDECAR_IMAGE" --format json \
  > reports/ordinary-inventory.json

Exit 0 means the reader verified the DAGs, original pack fingerprints, detached runtime archives, dependency closure, and supported producer reconstruction. Retain inventory_sha256. Inventory does not publish or rewrite its source root. The sidecar is validated here, but its choice is not part of the source inventory digest.

Generate composition.yaml from the actual reports, rather than copying example digests. Run this from the repository root, after verifying the inventory command succeeded:

import json
import os
from pathlib import Path

import yaml

native = json.loads(Path("reports/native-materialize.json").read_text())
ordinary = json.loads(Path("reports/ordinary-inventory.json").read_text())
assert native["passed"] and ordinary["passed"]
manifest = {
    "schema": "dpone.release-composition.v1",
    "native_workspace": {
        "root": str(Path(native["release_dir"]).absolute()),
        "expected_release_id": native["release_id"],
    },
    "standalone": {
        "root": ".dpone/ordinary-build/airflow",
        "expected_inventory_sha256": ordinary["inventory_sha256"],
    },
    "transport": {
        "profile": "compact_v2_runtime_connection_context",
        "xcom_sidecar_image": os.environ["XCOM_SIDECAR_IMAGE"],
    },
}
Path("composition.yaml").write_text(yaml.safe_dump(manifest, sort_keys=False))

dpone gitops airflow release-compose \
  --manifest composition.yaml --output-dir composed-release --format json \
  > reports/composition.json

--output-dir is the exact immutable destination, not a cache root. Relative input roots are resolved from composition.yaml; the output path is resolved from the command's working directory. All three roots must be disjoint. The manifest and reports belong outside the immutable destination.

On exit 0, the report has passed: true, status: passed, the parent release_id, the native source_release_id, and the ordinary inventory_sha256. composed-release/release-set.json contains the explicit v3 envelope. Its dags/, packs/, native runtime payloads, _composition/ source artifacts, and release-subjects.sha256 form one immutable tree. No illustrative output here substitutes for inspecting your own report.

Use the same service from Python

The Python entrypoint uses the same verifiers and publication semantics as CLI:

from pathlib import Path

from dpone.app.release_composition import build_release_composition_service
from dpone.manifest.release_composition_request import read_release_composition_request

service = build_release_composition_service()
request = read_release_composition_request(
    Path("composition.yaml"), output_dir=Path("composed-release")
)
report = service.compose(request)
if not report.passed:
    raise RuntimeError(report.to_dict())
print(report.release_id)

An identical retry verifies the same immutable destination. Never treat durability_uncertain as success. Follow delivery and recovery to install the complete parent into a cache and prepare its deployment projection.