Feature design: compact pack → v2 RuntimeConnectionContext promotion¶

Status: APPROVED
Implementation: OSS #430 landed in branch; live dual-load retirement evidence UNVERIFIED pending master publish (see test_artifacts/connection-ref-smoke-pilot/20260723T053500Z-dual-load-retirement-status.json)
Owner: dpone maintainers
Issue: retire smoke-v2 dual-load after sealed connection_ref cutover
Target release: next patch after 0.73.x tip that includes #427
Last verified: 2026-07-23
Revision: fills the compact-pack gap left open by docs/feature-design-airflow-init-fetch-connection-bridge-v0732.md (which correctly kept compact-only context as a non-goal) by promoting compact reconcile outputs onto the existing v2 init_fetch authority
Architecture authority: ADR 0010, ADR 0027, ADR 0009 / 0024, docs/airflow-self-service-architecture.md

Executive summary¶

DEV sealed kubernetes_secret_volume globally and proved connection_ref pilots GREEN, but the scheduler still dual-loads a smoke-v2 airflow-index.v2 ahead of compact repo-root packs because only the v2 path delivers verified RuntimeConnectionContext.

This slice does not invent a second context authority inside RuntimeConnectionContextLoader or the legacy pack-index.json lane. It promotes compact reconcile packs into an immutable dpone.release-set.v1 (with the same closed strict rewrite consumers already apply in materialize_smoke_v2_release.py), then reuses AirflowDeploymentProjectionService.materialize + dpone airflow build/publish/cache-sync so the single scheduler load path is a v2 index with binding-set / rewritten connection-registry / credential-runtime descriptors.

Success: consumer removes dual-load / PILOT_DAGS special casing; marketing, one interchange DAG, and kam_mes succeed without the second compact load.

Personas and customer journey¶

Persona	Goal	Current pain	Success signal
Platform engineer	One publish/load path for sealed `connection_ref`	Dual-load + hardcoded pilot list	Single v2 index load
Pipeline author	Keep domain YAML / sealed projection	Must know smoke-v2 dual-load exists	No dual-load docs in cutover
Airflow operator	Keep KAM/S3 stage green	Context missing → Airflow env fallback	Context-only S3 resolve

Journey:

dpone gitops airflow reconcile
dpone gitops airflow release-materialize (new) → release-set
Existing dpone airflow build / publish / cache-materialize / cache-sync
Scheduler: one load_dpone_dags(index_path=.../current/airflow-index.json)
Runtime: init_fetch plan + DPONE_RUNTIME_CONNECTION_CONTEXT (unchanged)

Scope¶

In scope¶

OSS glue: compact pack-root → immutable release-set with closed strict rewrite
Auto-discovery of dag-specs/workloads from reconcile output (no hardcoded pilots)
Fail-closed rewrite for sealed bridge projection + digest-pinned XCom sidecar
Docs / changelog / consumer follow-up notes
Focused unit tests for materialize + rewrite

Non-goals¶

Second RuntimeConnectionContext authority without init_fetch plan pins
Embedding secrets or registry bodies into pack-index.json
Global flip back to unsafe_airflow_env
Changing KAM/S3 stage semantics beyond using existing context preference
Full remote pack-index deprecation for unrelated platform DAGs

Assumptions and constraints¶

Loader still requires plan env + context path (ADR 0027 fail-closed).
Closed bridge projection remains complementary to context mounts.
Consumer domain fleet currently covered by the former pilot set can move to a single v2 index without losing scheduled domain DAGs.

Public contract¶

CLI¶

dpone gitops airflow release-materialize
  --pack-root .dpone/gitops/airflow
  --cache-root .dpone-cache
  --xcom-sidecar-image <digest-pinned OCI ref>
  [--dag-id DAG...]                 # optional subset; default = all dag-specs
  [--output PATH] [--format json|markdown]

Exit 0 on success, 2 on blockers. JSON report includes release_id, dag_ids, workload_ids, pack_fingerprints, connection_projection_mode, strict_init_fetch_rewrite.

Python API¶

from dpone.readiness.airflow_compact_pack_release import materialize_compact_pack_release

report = materialize_compact_pack_release(
    pack_root=Path(".dpone/gitops/airflow"),
    cache_root=Path(".dpone-cache"),
    xcom_sidecar_image="harbor.../alpine@sha256:...",
)

Manifest/schema¶

No new public schemas. Reuses dpone.release-set.v1 and existing v2 deployment/index schemas with runtime_artifact_delivery.mode=init_fetch.

Artifacts and evidence¶

Release tree: .dpone-cache/releases/sha256-<digest>/
Downstream: existing deployment projection writes cache://runtime-connection-contexts/sha256-…/{binding-set,connection-registry,credential-runtime}.json

Compatibility and migration¶

Legacy gitops airflow publish pack-index lane unchanged (no context).
Consumers replace materialize_smoke_v2_release.py with this CLI.
Dual-load may remain only until the scheduler points at one v2 current.

Detailed algorithm¶

Discover pack_root/_dags/*.dag-spec.json (or filter by --dag-id).
For each dag-spec, collect workload_id values from nodes; require each pack_root/<workload_id>/airflow-pack.json.
Rewrite each pack:
keep closed connection_projection (kubernetes_secret_volume, payload_format=airflow_connection_uri, secret_values=false);
preserve URI override maps (query_overrides / scheme_overrides / database_overrides);
strip non-closed airflow.execution extras;
pin xcom.sidecar_image;
inject bounded artifact-reader env templates into provider_execution.kpo_kwargs.env_vars;
recompute pack_fingerprint via compute_pack_fingerprint.
Rewrite each dag-spec: drop operator_overrides; recompute dag-spec fingerprint.
Build dpone.release-set.v1, compute release_id, write immutable release via materialize_immutable_local_release.
Fail closed on missing packs, open projection mode, fingerprint drift, or empty selection.
Consumer continues with existing airflow build (projection + context snapshots) → publish → cache-sync.

Pseudocode¶

dags = discover_dag_specs(pack_root) filtered by dag_ids
for dag in dags:
  packs[workload] = rewrite_strict(load_pack(workload))
  specs[dag.id] = rewrite_dag_spec(dag)
release = assemble_release_set(dags, packs)
release_id = compute_release_id(release)
materialize_immutable_local_release(release_dir, files)

Edge cases¶

Case	Behavior
Empty pack-root / no dag-specs	blocker, exit 2
Missing workload pack	blocker naming workload id
Projection not sealed	blocker
Duplicate release bytes	immutable materializer create-or-compare
Compact publish without this step	still no RuntimeConnectionContext

Architecture¶

Components and responsibilities¶

Component	Existing/new	Responsibility
`readiness.airflow_compact_pack_release`	new	Strict rewrite + release-set assembly
`AirflowDeploymentProjectionService`	existing	Context snapshots + v2 index
`RuntimeConnectionContextLoader`	existing	Unchanged fail-closed authority
Consumer loader	consumer	Single v2 index load

Alternatives and tradeoffs¶

Alternative	Decision
Embed context into pack-index.json	Reject — second authority / ADR conflict
Teach loader to trust bare context path	Reject — breaks plan pin
Keep dual-load forever	Reject — operational debt

ADR requirement¶

No new ADR. Extends ADR 0009/0024 delivery and ADR 0010/0027 authority by promotion, not a parallel model.

Quality-budget impact¶

One new gitops module (<250 SLOC target), thin command/service wrappers. No god-module split required.

Market comparison¶

System	Relevant capability	Adopt/reject	Source/date
dlt	N/A — no Airflow pack/init_fetch authority model	N/A	—
Airbyte	Connector config delivery ≠ verified runtime context	Reject parallel	airbyte.com docs 2026-07
Fivetran	Managed creds, not OSS pack promotion	N/A	—
Astronomer Cosmos	dbt task generation, not connection context packs	N/A	—
gusty	DAG generation from YAML	Reject as context authority	gusty docs 2026-07
Informatica / Pentaho / SSIS / Beam	N/A for this Airflow pack lane	N/A	—

Measurable differentiation¶

axis: single verified connection authority at Airflow parse/runtime
scenario: sealed connection_ref DAG without smoke-v2 dual-load
baseline: dual-load compact + smoke-v2 index
metric: representative DAG SUCCESS with loader dual-load removed
target: marketing + one interchange + kam_mes SUCCESS
procedure: pin OSS → consumer MR → publish v2 → sync → trigger
artifact: test_artifacts/connection-ref-smoke-pilot/*-dual-load-retired*.json
limitations: platform Python DAGs outside gitops domains remain separate modules

Security, privacy, and operations¶

No secret values in release-set or index.
Registry rewrite remains mount-backed kubernetes_secret_volume.
Attempt Secret RBAC stays a consumer prerequisite.
Logs/evidence expose only logical refs and digests.

Test and certification plan¶

Layer	Scenario	Expected
Unit	rewrite preserves URI overrides	PASS
Unit	release_id stable / drift fails	PASS
Unit	open projection blocked	PASS
Contract	release feeds projection context descriptors	PASS (reuse existing projection tests)
Live	marketing / interchange / kam_mes without dual-load	PASS or SKIP with reason

Documentation plan¶

This feature design
docs/airflow-connection-ref-consumer-followups.md update
CHANGELOG
Consumer cutover docs (consumer repo)

Rollout and rollback¶

Land OSS CLI + pin consumer.
Switch publish script to OSS materialize; keep same registry URI initially.
Loader: single v2 index; delete dual-load.
Rollback: restore dual-load loader commit; repoint cache to prior deployment.

Agent execution plan¶

Agent/role	Owned paths	Forbidden
Integrator	feature design, gitops release module, command, tests, changelog, followups	unrelated connectors
Consumer follow-up	airflow-dags-dev loader/CI/docs	OSS secrets

Approval checklist¶

User problem and CJM are clear.
Algorithm and failure semantics are implementable without guessing.
Public contracts and compatibility are explicit.
Architecture and alternatives are justified.
Relevant market research uses current official sources.
Claimed differentiation is measurable.
Tests, evidence, docs, rollout, and rollback are complete.
Path ownership and integration plan are conflict-safe.
Maintainer instruction ("Do this") + ADR 0010/0027 / self-service gap authorize APPROVED for this vertical slice.