Skip to content

GitOps workload catalog

Advanced path (not the beginner journey). If you are creating your first Airflow pipeline, start at First Airflow DAG instead.

dpone gitops workloads is the high-level GitOps facade for large Airflow repositories. It lets teams describe many workloads declaratively, resolve environment/domain/source overrides, and generate one compact Airflow pack per workload.

Use it when .dpone/gitops and CI started accumulating repeated generated files or long chains of low-level dpone gitops airflow ... commands. See the advanced Data Engineer CJM for when this belongs in your journey.

Workload set

gitops:
  version: 1

  defaults:
    runner: airflow
    resources_profile: safe_worker
    outcome_mode: xcom_then_gate

  environments:
    dev:
      namespace: airflow-dwh-intgr-test-1
      runner_policy: advisory
    prod:
      namespace: airflow-dwh-prod
      runner_policy: certified_only

  source_types:
    mssql:
      resources_profile: throughput

  sources:
    mssql_dwh_reporting:
      source_type: mssql
      snapshot_profile: weak_worker_chunks

  includes:
    - path: gitops/domains/**/*.yaml
    - path: gitops/environments/**/*.yaml
    - manifest_glob: manifests/**/*.yaml
      infer_workload: true

Catalog files are included by the root workload set. Directory names are only organization; semantics come from typed fields such as domain, environment, defaults, and workloads.

domain: interchange
defaults:
  owner: data-office
  labels: [interchange]

workloads:
  inter_ch_kontr_inn_for_email:
    manifest: ../../manifests/mssql/interchange/kontr_inn_for_email.yaml
    source: mssql_dwh_reporting
    schedule: "0 7 * * *"
    resources_profile: safe_worker

Override precedence

Effective config is resolved in this order:

global < environment < runner < source_type < source < sink_type < sink < domain < workload < manifest-local < CLI

Every effective value carries provenance, so workloads explain can show where the value came from.

Compact Airflow runner contract

See gitops-airflow-runner-contract.md for the full industrial contract (airflow.runner: placement, embed assets, connection binds, reconcile-time validation).

environments:
  prod:
    namespace: airflow-dwh
    airflow:
      runner:
        placement:
          node_selector:
            dedicated: datawarehouse
          tolerations:
            - key: dedicated
              operator: Equal
              value: datawarehouse
              effect: NoSchedule
        embed_assets:
          - path: certs/RootCA.pem
            bind:
              connection_id: ClickHouseCloud_marketing
              query_key: ca_cert
      connection_projection:
        mode: unsafe_airflow_env
        connection_ids:
          - ClickHouseCloud_marketing

dpone gitops airflow reconcile writes placement into pod_spec, embeds assets into the inline archive, and expands bind / workspace_query_overrides into connection_projection.query_overrides.

Commands

dpone gitops workloads list \
  --workload-set dpone_workloads/gitops.yaml \
  --env dev

dpone gitops workloads explain inter_ch_kontr_inn_for_email \
  --workload-set dpone_workloads/gitops.yaml \
  --env dev

dpone gitops airflow pack \
  --workload inter_ch_kontr_inn_for_email \
  --workload-set dpone_workloads/gitops.yaml \
  --env dev \
  --output-path .dpone/gitops/airflow/inter_ch_kontr_inn_for_email/airflow-pack.json

dpone gitops airflow reconcile \
  --workload-set dpone_workloads/gitops.yaml \
  --changed-files-file changed.txt \
  --env dev \
  --output-dir .dpone/gitops

dpone gitops gitlab render-child-pipeline \
  --workload-set dpone_workloads/gitops.yaml \
  --changed-files-file changed.txt \
  --env dev \
  --output .dpone/gitops/child.yml

airflow pack writes one self-contained airflow-pack.json with workload identity, effective config, runtime command, pod spec, optional connection projection policy, XCom sidecar pinning, outcome gate, runtime evidence paths, and pack fingerprint. Airflow DAG files should use the thin pack facade instead of reading many generated files such as kpo-kwargs.json, pod-spec.yaml or run-spec.json.

from dpone_airflow_pack import build_dpone_gitops_task_group_from_pack

build_dpone_gitops_task_group_from_pack(
    "/opt/airflow/dags/.dpone/gitops/airflow/inter_ch_kontr_inn_for_email/airflow-pack.json"
)

CI pattern

For small repositories, one CI job can run dpone gitops airflow reconcile. For large repositories, use dpone gitops gitlab render-child-pipeline to generate one child job per affected workload. This keeps .gitlab-ci.yml short while preserving parallel validation.

The low-level commands such as run-spec, runtime-profile, pod-contract, artifact-index, and preflight remain available for debugging and compatibility. New repositories should prefer the workload catalog facade.

Guardrails

  • All evidence paths are repo-relative.
  • Secret values must not appear in workload catalogs or generated packs.
  • Production policy can be tightened by environment defaults; weakening it should be explicit and reviewable.
  • Airflow scheduler parse should read static pack JSON only and must not run GitOps rendering at import time.