Skip to content

Gitops airflow runner contract

Compact Airflow runner contract

Environment and domain defaults configure Kubernetes placement and runner-local assets through a single airflow.runner block. This mirrors how mature platforms split concerns:

Platform Placement Runner-local files Secret/path rewrite
Airflow Helm chart tolerations / nodeSelector in values per env git-sync volume on workers Connections in K8s Secret / Vault
Argo Workflows podSpecPatch / affinity per template artifacts / init containers Parameter + volume mounts
Airbyte worker node pools / tolerations connector-specific mounts in job pod secret hydration at runtime
dpone compact pack runner.placement → pod_spec runner.embed_assets → inline tarball bind + workspace_query_overrides → query_overrides

KPO pods created by compact packs do not mount git-synced DAG folders. Any TLS root, JDBC driver, or license file referenced by a connection URI must therefore be declared as a runner embed asset.

environments:
  prod:
    namespace: airflow-dwh
    airflow:
      runner:
        workspace_root: /workspace/repo
        placement:
          node_selector:
            dedicated: datawarehouse
          tolerations:
            - key: dedicated
              operator: Equal
              value: datawarehouse
              effect: NoSchedule
            - key: dedicated
              operator: Equal
              value: datawarehouse
              effect: NoExecute
        embed_assets:
          - path: certs/RootCA.pem
            bind:
              connection_id: ClickHouseCloud_marketing
              query_key: ca_cert
      connection_projection:
        mode: unsafe_airflow_env
        connection_ids:
          - ClickHouseCloud_marketing
          - mssql_prod_tech_user_pyton

dpone gitops airflow reconcile applies the contract to every compact pack:

  • runner.placement is written into pod_spec.spec.nodeSelector and pod_spec.spec.tolerations.
  • runner.embed_assets / legacy runner.embed_paths are bundled into the inline workload tarball at /workspace/repo/....
  • bind and legacy workspace_query_overrides expand repo-relative paths into absolute runner paths in connection_projection.query_overrides.
  • Missing embed assets fail reconcile with runner_embed_asset_missing instead of a silent runtime TLS failure inside the pod.

Legacy flat keys (airflow.node_selector, airflow.tolerations, airflow.runner_embed_paths) remain supported for existing workload sets.