Skip to content

Domain-first Airflow project

Use domain-first layout when pipeline ownership should be visible in the repository tree and a team should be able to add a pipeline without editing a shared domain catalog.

This journey creates one domain and one MSSQL to ClickHouse pipeline. It uses only authoring files, performs no network or secret-store calls, and does not require Airflow Python.

Before you start

This page is for a data engineer creating a first domain-owned pipeline. Follow the installation guide, verify dpone --version, and use a new empty working directory. The feature is currently documented under Unreleased; until its patch release is published, install the reviewed source checkout explicitly:

export DPONE_REVIEWED_REVISION="<exact-release-tag-or-40-character-commit>"
test "$DPONE_REVIEWED_REVISION" != "<exact-release-tag-or-40-character-commit>"
git clone --no-checkout https://github.com/PaulKov/dpone.git
cd dpone
git fetch origin "$DPONE_REVIEWED_REVISION"
git checkout --detach "$DPONE_REVIEWED_REVISION"
test "$(git rev-parse HEAD)" = "$(git rev-parse "$DPONE_REVIEWED_REVISION^{commit}")"
uv sync --frozen
export PATH="$PWD/.venv/bin:$PATH"
dpone init domain --help
mkdir ../dpone-domain-first-quickstart
cd ../dpone-domain-first-quickstart

The release or change owner supplies the exact reviewed revision; do not use a moving branch name for this source-checkout path.

After the patch is published, the normal pip install dpone path replaces these source-checkout steps. A supported build exposes dpone init domain --help and --layout domain-first. Run every journey command from the new empty quickstart directory. The route is credential-free until a platform engineer binds the generated logical connection references.

Five-command journey

The product target is completion in 15 minutes or less after installation and access to an empty project directory. Automated tests prove that the commands and artifacts work, but they are not human-usability evidence. The target remains UNVERIFIED until a milestone study records at least five new users and at least 80% finish without help.

dpone init project --airflow --layout domain-first
dpone init domain crm \
  --owner-team data-crm \
  --owner-contact crm@example.com \
  --approver-team data-platform
dpone init pipeline orders_daily \
  --domain crm \
  --route mssql:clickhouse:incremental_merge \
  --from mssql_dev:dbo.orders \
  --to clickhouse_dev:analytics.orders \
  --key order_id
dpone check crm/orders_daily
dpone airflow preview orders_daily

Every command returns exit code 0. The two validation commands finish with:

dpone check: OK
dpone airflow preview: OK
- deployment: preview (not runnable)
- airflow index: .dpone-cache/current/airflow-index.json

Any non-zero exit stops the journey. Follow the structured error's docs and fix lines before continuing.

Run the generated credential-free behavior test when you want an executable local proof:

dpone test orders_daily

The route command resolves through the same capability catalog used by dpone recipe list; it is not a second recipe table. Unsupported routes fail before any file is written.

Files you own

The project contains:

dpone.yaml
workloads/
  crm/
    ownership.yaml
    pipelines/
      orders_daily/
        pipeline.yaml
        tests/
          pipeline.test.yaml
          fixtures/
            input.jsonl
dags/
  dpone.py
environments/
  dev/
    binding-set.yaml
    credential-runtime.yaml
platform/
  connection-registries/
    dev.yaml
Path Responsibility
dpone.yaml Selects domain_first, the confined authoring root, and project-wide pipeline ids.
workloads/crm/ownership.yaml Domain owner and approver metadata.
workloads/crm/pipelines/orders_daily/pipeline.yaml The only editable source for this pipeline.
Colocated tests/ Hermetic behavior contract and fixture.
dags/dpone.py Generated provider loader; commit it but do not edit it.
environments/dev/ Non-secret development binding and credential-runtime references.
platform/connection-registries/dev.yaml Non-secret logical connection registry stub.

Do not create or edit domains/*.yaml in a domain-first project. Do not commit .dpone-cache/** or a generated workload index.

Preview creates immutable release/deployment artifacts and the local current pointer below .dpone-cache/. They are generated, content-addressed, and must not be edited or committed.

Project-wide operations

Target one pipeline by id or by domain-qualified shorthand:

dpone check orders_daily
dpone check crm/orders_daily

Select all pipelines in a domain for project-wide validation or preview:

dpone check . --select 'domain:crm'
dpone airflow preview . --select 'domain:crm'

Selection consumes the same discovery snapshot used by individual lookup and immutable preview materialization. A pipeline is not rediscovered by a different algorithm for build or publish.

Next steps