Airflow cache sync and recovery¶

Use this runbook when a platform deployment step must make one already built dpone deployment visible to the Airflow loader. cache-sync performs local, fail-closed verification and promotion. It never downloads artifacts, rebuilds a release, reads credentials, or calls Airflow, Vault, Kubernetes, or a remote registry.

Prerequisites¶

The service identity running the command needs read access to the candidate deployment and pinned release, plus write access to the cache pointer and audit files. The cache must contain all of the following:

Use two explicit roots in automation:

DPONE_BUILD_CACHE_ROOT="${DPONE_BUILD_CACHE_ROOT:-${PWD}/.dpone-cache}"
DPONE_SCHEDULER_CACHE_ROOT="${DPONE_SCHEDULER_CACHE_ROOT:-/opt/airflow/dags/.dpone-cache}"

DPONE_BUILD_CACHE_ROOT is the CI build/publish input. The materializer writes the exact pinned projection to DPONE_SCHEDULER_CACHE_ROOT, and every promotion, recovery, retention, and provider path below uses that scheduler root. Do not let a relative build cache accidentally become the scheduler cache.

.dpone-cache/
  releases/sha256-<release>/
    release-set.json
    dags/...
    packs/...
  deployments/<environment>/sha256-<deployment>/
    deployment.json
    airflow-index.json
    _SUCCESS

release-set.json uses release-relative path values. airflow-index.json uses pinned cache://releases/<release>/... references. Both must name the same DAG specs and workload packs with the same SHA-256 digests.

Before upgrading an older cache that retained only deployment projections, rematerialize the corresponding immutable release-set and artifacts through the platform publisher/materializer that originally created the release. There is no cache-sync restore mode: stop if that trusted input is unavailable. For a local preview, dpone airflow preview <pipeline> materializes the release and preview deployment. Production promotion remains a platform/CI operation. Generated Phase 1A/1B release-sets use the preferred path field. The v1 compatibility adapter also accepts the legacy field name artifact_ref when its value is the same release-relative path. Exactly one locator is required. cache:// references are invalid in a release-set because they would make its content identity self-referential; new producers must emit path.

--promoted-by and every --allowed-promoter value are policy inputs, not proof of caller identity. The platform must authenticate the process through CI/workload identity and restrict cache writes with filesystem permissions. The mutation CLI requires an allowlist so an accidental or misconfigured actor cannot bypass the policy check, but an untrusted process must never be allowed to choose both values and write the cache.

The readiness mutation facade applies the same rule: an empty or mismatched allowlist returns DPONE_CURRENT_POINTER_PROMOTER_UNAUTHORIZED before projection validation or cache mutation. Beginner commands do not expose actor flags. dpone airflow preview and the local safe-sample deployment path each use one fixed local actor as both the declared actor and the sole allowed actor. They observe the current physical deployment identity and submit either an expected-current or expected-absent CAS guard. A concurrent change therefore returns DPONE_CURRENT_POINTER_CAS_MISMATCH; rerun the original beginner command to evaluate the newly active deployment instead of retrying a stale promotion automatically.

Python callers of the readiness facade must now provide the platform-owned allowlist explicitly:

result = cache_sync_result(
    cache_root=cache_root,
    deployment_dir=deployment_dir,
    environment="prod",
    promoted_by=authenticated_actor,
    allowed_promoters=platform_allowed_actors,
    confirm_promote=True,
)

Calls that omit the policy, or whose actor is not an exact allowlist member, return exit code 4 and do not mutate the cache. Build the allowlist from trusted deployment configuration, never from the same untrusted request that supplies promoted_by. The CLI and five-command beginner journey require no migration.

Recognize provider parse failures¶

The scheduler-side LoadReport separates a shared index failure from damage to one listed artifact:

Signal	Meaning	Operator response
`fatal: true`	The provider could not establish a trusted deployment index; no DAG from it is safe to load.	Keep the generated loader guard enabled, preserve the first error code, and inspect cache state before mutation.
`fatal: false` with `errors[].dag_id`	The index is trusted, but one DAG spec or workload pack failed its isolated size/digest/read/materialization check.	Leave unrelated valid DAGs available, but rematerialize the affected immutable release/projection.

For a local authoring preview, regenerate the complete local projection:

dpone airflow preview orders_daily
dpone airflow explain orders_daily

For a shared environment, do not retry cache-sync blindly. Start with the read-only planner and inspect status, issues, current_path_deployment_id, and preferred_repair_deployment_id:

dpone airflow cache-recovery-plan \
  --cache-root /opt/airflow/dags/.dpone-cache \
  --environment prod \
  --format json

If the plan identifies missing or corrupt immutable content, rematerialize the exact release/deployment from trusted storage, rerun the plan, and apply only its reviewed recovery or promotion action. Never hand-edit the active airflow-index.json, DAG spec, or workload pack.

Materialize the prerequisite¶

Build the environment projection after CI has produced the environment-neutral release. Set real canonical digests first; the fail-fast shell checks prevent a placeholder from becoming a deployment. This is the production parse-canary prerequisite: the production example below prepares a paused parse canary, not permission to trigger tasks.

: "${DPONE_RELEASE_ID:?set the canonical release sha256 digest}"
: "${DPONE_RUNTIME_IMAGE_DIGEST:?set the canonical runtime image sha256 digest}"
: "${DPONE_REGISTRY_CONFIG_SHA256:?set the registry ConfigMap file sha256 digest}"
: "${DPONE_TRUST_POLICY_SHA256:?set the trust-policy ConfigMap file sha256 digest}"
: "${DPONE_SOURCE_COMMIT:?set the Airflow bundle Git commit}"

dpone airflow build \
  --release-id "${DPONE_RELEASE_ID}" \
  --environment prod \
  --trust-tier production \
  --runtime-image-ref "registry.example/dpone-runtime@${DPONE_RUNTIME_IMAGE_DIGEST}" \
  --runtime-image-digest "${DPONE_RUNTIME_IMAGE_DIGEST}" \
  --artifact-registry-ref dpone-prod-artifacts \
  --registry-config-map-name dpone-artifact-registry \
  --registry-config-map-key registry.json \
  --registry-config-sha256 "${DPONE_REGISTRY_CONFIG_SHA256}" \
  --trust-policy-config-map-name dpone-artifact-trust-policy \
  --trust-policy-config-map-key policy.json \
  --trust-policy-sha256 "${DPONE_TRUST_POLICY_SHA256}" \
  --airflow-bundle-ref "git:${DPONE_SOURCE_COMMIT}"

The command emits a structurally executable dpone.airflow-deployment-index.v2. It binds the exact digest-pinned runtime image, explicit trust tier, registry ConfigMap snapshot, workload identity, and production trust-policy snapshot. In the stock candidate, production remains a parse-only canary because no concrete attestation verifier is configured. If the installed producer does not expose these options, stop and upgrade it; never patch a v1 index into v2.

Publish that exact release/deployment pair. This command validates every local fingerprint and checksum before constructing the optional storage client. It uses conditional create-or-compare and publishes each _SUCCESS marker last; it never overwrites or deletes a content-addressed object:

: "${DPONE_DEPLOYMENT_ID:?set the deployment sha256 digest emitted by build}"

dpone airflow publish \
  --cache-root "${DPONE_BUILD_CACHE_ROOT}" \
  --release-id "${DPONE_RELEASE_ID}" \
  --deployment-id "${DPONE_DEPLOYMENT_ID}" \
  --environment prod \
  --artifact-registry-ref dpone-prod-artifacts \
  --registry-uri s3://platform-artifacts/dpone/airflow \
  --identity-mode workload_identity \
  --format json

Run materialization as a deployment step, init container, or sidecar. Supply both exact identities; current, latest, remote listing, tags, and branch names are not accepted:

dpone airflow cache-materialize \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --release-id "${DPONE_RELEASE_ID}" \
  --deployment-id "${DPONE_DEPLOYMENT_ID}" \
  --environment prod \
  --artifact-registry-ref dpone-prod-artifacts \
  --registry-uri s3://platform-artifacts/dpone/airflow \
  --identity-mode workload_identity \
  --max-object-bytes 67108864 \
  --max-total-bytes 536870912 \
  --format json

For Azure Workload Identity, install dpone[azure] or dpone[object_storage] and use the account-scoped form:

dpone airflow cache-materialize \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --release-id "${DPONE_RELEASE_ID}" \
  --deployment-id "${DPONE_DEPLOYMENT_ID}" \
  --environment prod \
  --artifact-registry-ref dpone-prod-artifacts \
  --registry-uri azure://platformaccount/dpone-artifacts/dpone/airflow \
  --identity-mode workload_identity

This mode constructs Azure WorkloadIdentityCredential only during command execution. The accountless az:// form remains available to injected clients, but is rejected for workload identity because no safe account URL can be derived from it. S3, GCS, and Azure registry roots must all include a non-empty canonical path after the bucket/container.

The materializer first pins the local cache-root identity, checks both remote completion markers, and streams only fixed deployment files and release-declared artifacts into a private mode-0700 operating-system staging directory through write-time byte limits. It verifies downloaded sizes and SHA-256 values, runs the full projection validator, and uses atomic no-replace installation for immutable local trees. It does not create or change current. A separate reviewed cache-sync remains the only activation step. The publisher similarly freezes every validated source file into a private, mode-0700 operating-system temporary snapshot before its first registry operation, so changing a build file during upload cannot change the bytes associated with a pinned identity and replacing the cache pathname cannot redirect snapshot bytes.

After promotion, the provider reads <cache-root>/current/airflow-index.json. It infers <cache-root> from that lexical pointer before resolving the activation symlink, so custom cache directory names retain correct cache://releases/... resolution.

The accepted provider pointer is deliberately narrower than a generic local symlink:

current -> activations/<environment>/sha256-<64 lowercase hex>

The deployment ID and, when present in the index, environment must match that activation identity. A direct directory or a symlink to deployments/** is not a compatibility path; run the reviewed materialize/promote workflow to create the sealed activation instead.

For a deterministic local proof, replace --identity-mode workload_identity with --local-registry-root .dpone-artifact-registry. publish may instead use a bounded logical --connection-id <logical-id> plus --connection-type when CI cannot use workload identity. The ID grammar is [A-Za-z0-9][A-Za-z0-9_.-]{0,127}; it is a name, never a URI, JSON payload, Vault path, token, or secret value. cache-materialize intentionally exposes no Airflow/Vault connection resolver options: a deployment step uses workload identity, while local proof uses the emulator root.

The local emulator records one root device/inode for the client lifetime and opens the root from the filesystem anchor and every descendant directory with descriptor-relative no-follow operations. A symlink in any root ancestor is rejected before a directory is created. Conditional create, stat, download, listing, and deletion remain anchored to the first root identity. Replacing the pathname during or between object operations fails closed, so artifacts and _SUCCESS cannot be split across different trees. Noncanonical root paths also fail closed.

Production deployment projections generated with verify.attestations: required_for_prod fail with DPONE_ARTIFACT_ATTESTATION_REQUIRED until a certified attestation verifier is available through a separately approved runtime composition. The stock runtime CLI in this patch has no concrete verifier configuration surface, so production execution is intentionally fail-closed and must not be promoted as runnable. There is no bypass flag. S3/GCS/Azure cloud-live and Kubernetes end-to-end certification remain UNVERIFIED unless the exact environment and credentials are available; local object-store tests are not a substitute.

Publication/materialization failure recovery is retry-by-revalidation:

Code	Meaning	Safe action
`DPONE_ARTIFACT_REGISTRY_INCOMPLETE`	Marker or declared object is absent.	Rerun the same pinned publisher; do not activate.
`DPONE_ARTIFACT_REGISTRY_IMMUTABILITY_CONFLICT`	Existing bytes differ at a content-addressed key.	Quarantine the prefix and investigate; never overwrite it.
`DPONE_ARTIFACT_REGISTRY_CHECKSUM_MISMATCH`	Downloaded bytes do not match the release contract.	Stop, audit storage, and create a new valid release.
`DPONE_ARTIFACT_REGISTRY_METADATA_INVALID`	Storage returned a negative or malformed object size.	Stop, inspect the backend/adapter, and rerun only after metadata is trustworthy.
`DPONE_ARTIFACT_REGISTRY_OBJECT_TOO_LARGE`	One object exceeds the local policy.	Review the artifact and explicit platform budget.
`DPONE_ARTIFACT_REGISTRY_TOTAL_LIMIT_EXCEEDED`	The bounded projection exceeds the total budget.	Review release size and platform limits.
`DPONE_ARTIFACT_REGISTRY_UNAVAILABLE`	Storage, IAM, or credential dependency is unavailable.	Repair the dependency and rerun the exact pins.

If a legacy production cache has no trusted release source, stop the upgrade and escalate to the platform artifact owner. Do not reconstruct an immutable release by editing deployment JSON or copying the currently visible pack.

Operate strict-v2 init-fetch pods¶

Cache promotion and pod execution are separate trust boundaries. cache-sync makes one verified index visible to the parse-safe provider; it does not contact Kubernetes, mount ConfigMaps, exercise workload identity, or certify runtime fetch.

Run an executable migration rehearsal only with trust_tier: non_production in an approved dev/stage environment. A production projection may be promoted to a paused production parse canary, but its tasks must not be triggered until the configured verifier and current live certification evidence exist.

Prepare the runtime inputs¶

Before promotion, verify that the executable deployment has:

schema: dpone.airflow-deployment-index.v2;
runtime_artifact_delivery.mode: init_fetch;
matching explicit top-level and delivery-level trust_tier;
an OCI runtime_image_ref pinned by the same digest as runtime_image_digest;
exact release, deployment, and selected workload descriptors with positive bytes;
kubernetes_workload_identity with the reviewed namespace and service account;
digest-pinned ConfigMap references for registry configuration and, for production, trust policy.

The registry configuration is runtime-only platform configuration. It may contain a physical registry URI but no static credential:

{
  "registries": {
    "dpone-prod-artifacts": {
      "access": {
        "mode": "workload_identity"
      },
      "registry_uri": "s3://platform-artifacts/dpone/airflow"
    }
  },
  "schema": "dpone.artifact-registry-runtime-config.v1"
}

A production trust policy has this closed shape:

{
  "attestations": "required_for_prod",
  "schema": "dpone.runtime-artifact-trust-policy.v1",
  "trust_tier": "production"
}

Publish each exact file as the selected ConfigMap key, preferably under a new immutable, content-addressed ConfigMap name. Compute SHA-256 over the exact mounted file bytes, including the final newline when present, and put that digest in the v2 reference. Do not put the URI, ConfigMap body, token, signed URL, secret, or Vault path in airflow-index.json.

At pod start, the init process:

resolves each selected registry.json / policy.json path under its mount directory (following kubelet ConfigMap projection symlinks such as key -> ..data/key), then opens the resolved regular file once with no-follow semantics;
reads at most 64 KiB from each descriptor;
verifies the digest over those exact bytes;
parses those same bytes;
builds the registry reader only after verification.

A symlink that escapes the mount directory is rejected before read.

The ConfigMap name is therefore a selector, not integrity evidence. A digest mismatch is not retryable as a transient registry outage. Roll out changed bytes as a new reviewed ConfigMap snapshot and immutable deployment identity; do not rely on an in-place ConfigMap update.

Verify the fixed pod contract¶

The provider must produce exactly these commands with no arguments:

Container	Image	Command
`dpone-runtime-init-fetch`	Exact `runtime_image_ref`	`dpone airflow runtime-init-fetch`
base	The same exact image	`dpone airflow runtime-pack-exec`

Reserved mounts are:

Volume	Init	Base
`dpone-fetched-artifacts`	RW `/var/lib/dpone/artifacts`	RO `/var/lib/dpone/artifacts`
`dpone-worktree`	RW `/workspace/repo`	RO `/workspace/repo`
`dpone-run-output`	absent	RW `/var/lib/dpone/run`
`dpone-artifact-registry-config`	RO `/etc/dpone/artifact-registry`	absent
`dpone-artifact-trust-policy`	RO `/etc/dpone/artifact-trust` when selected	absent

Image, namespace, service account, security context, init containers, and these volumes/mounts are provider-owned. A pack collision must fail DPONE_INIT_FETCH_RESERVED_COLLISION; it must not be merged or ignored. Strict tasks also require retries=0; a non-zero DAG, pack, or override value fails DPONE_AIRFLOW_RETRIES_REQUIRE_TARGET_FENCE before any DAG is installed. This prevents automatic replay across an unresolved target-commit boundary but does not make a manual rerun idempotent.

The init container publishes /var/lib/dpone/artifacts/runtime-fetch-ready.json only after the plan, configuration, artifact bytes, deployment receipt, inventory, pack fingerprint, and effective attestation policy pass. The base container does not start when init fails. On start, runtime-pack-exec revalidates the same plan, ready manifest, deployment-set.v2 runtime receipt, independently verified release-set.v1, fetched pack, pinned trust-policy fingerprint, and effective attestation decision, then selects a structured argv from that verified pack. The base container does not mount or reread registry.json or policy.json; those are init-only inputs whose verified decision is committed by runtime-fetch-ready.json. Runtime workloads run the verified argv without shell=True, /bin/sh -c, scheduler-copied command, or inline bootstrap fallback; stdout/stderr are captured under /var/lib/dpone/run, and runtime-pack-exec always writes a JSON object to /airflow/xcom/return.json before exiting 0 so the KPO xcom sidecar can publish the outcome summary. Pre-hook selections still replace the process via os.execvpe.

Diagnose and recover¶

Interpret runtime command exits with the stable code in the container log:

Exit	Error family	Safe response
`0`	Init published a ready manifest.	Confirm the base container started; this is not route or data-success evidence.
`2`	`DPONE_INIT_FETCH_PLAN_INVALID`, `DPONE_INIT_FETCH_PLAN_TOO_LARGE`, `DPONE_INIT_FETCH_PLAN_HASH_MISMATCH`, or `DPONE_INIT_FETCH_PLAN_NON_CANONICAL`.	Rebuild the v2 index/KPO with compatible producer and provider bytes. Do not retry the same malformed pod.
`3`	`DPONE_ARTIFACT_REGISTRY_UNAVAILABLE`.	Repair registry/IAM availability, then retry the same pinned task. Each retry gets a fresh `emptyDir`.
`4`	Configuration/trust mismatch, missing or corrupt artifact, required attestation, invalid ready state, or other integrity/contract failure.	Stop execution. Restore exact immutable bytes or publish and promote a new deployment; do not bypass verification.
`5`	`DPONE_RUNTIME_PACK_EXEC_FAILED` or verified child process could not start.	Verify the digest-pinned runtime image contains the expected dpone entry point and runtime payload; rebuild the image/deployment if necessary.
Child exit	Verified workload command started and returned non-zero.	Use normal dpone run/evidence/state recovery for that exact attempt; the launcher preserves the child exit code.

Useful stable runtime codes include DPONE_ARTIFACT_REGISTRY_CONFIG_INVALID, DPONE_ARTIFACT_REGISTRY_CONFIG_MISMATCH, DPONE_ARTIFACT_TRUST_POLICY_INVALID, DPONE_ARTIFACT_TRUST_POLICY_MISMATCH, DPONE_ARTIFACT_ATTESTATION_REQUIRED, DPONE_CACHE_ARTIFACT_NOT_FOUND, DPONE_CACHE_ARTIFACT_TOO_LARGE, DPONE_RUNTIME_FETCH_READY_INVALID, and DPONE_RUNTIME_ARTIFACT_INTEGRITY_FAILED. Preserve the code and logical artifact reference. Do not copy physical URIs, mounted config bodies, tokens, or absolute pod paths into tickets or XCom.

Trust and certification status¶

The current TCB includes the producer, promoted cache/index, Airflow scheduler/provider, Kubernetes control plane and kubelet, digest-pinned runtime image, workload-identity/IAM enforcement, registry adapter, and configured attestation verifier. Fixed KPO fields constrain pack-owned overrides; they do not provide compromised-scheduler, compromised-provider, or compromised-cache resistance.

Unit and contract evidence is not live certification. Kubernetes projected ConfigMaps, workload identity, object storage, production attestation, Vault, MSSQL, and ClickHouse remain UNVERIFIED unless the exact commit, image digest, cluster, configuration, and evidence artifacts were exercised and retained.

Promote¶

For the first promotion, assert that no current deployment exists. Omitting the guard cannot protect two racing first promotions:

: "${DPONE_SCHEDULER_CACHE_ROOT:?set the absolute scheduler cache root}"
: "${DPONE_FIRST_DEPLOYMENT_DIR:?set the reviewed sha256-... deployment directory name}"

dpone airflow cache-sync \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --deployment-dir "${DPONE_SCHEDULER_CACHE_ROOT}/deployments/prod/${DPONE_FIRST_DEPLOYMENT_DIR}" \
  --environment prod \
  --promoted-by ci://your-platform/dpone-airflow \
  --allowed-promoter ci://your-platform/dpone-airflow \
  --expect-current-absent \
  --confirm-promote

For a later update, read current_path_deployment_id from dpone airflow cache-recovery-plan --environment prod --format json, review it, and use it as the CAS guard:

: "${DPONE_NEXT_DEPLOYMENT_DIR:?set the reviewed sha256-... deployment directory name}"
: "${DPONE_REVIEWED_CURRENT_DEPLOYMENT_ID:?set the current canonical sha256 digest}"
: "${DPONE_SCHEDULER_CACHE_ROOT:?set the absolute scheduler cache root}"

dpone airflow cache-sync \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --deployment-dir "${DPONE_SCHEDULER_CACHE_ROOT}/deployments/prod/${DPONE_NEXT_DEPLOYMENT_DIR}" \
  --environment prod \
  --promoted-by ci://your-platform/dpone-airflow \
  --allowed-promoter ci://your-platform/dpone-airflow \
  --expected-current-deployment-id "${DPONE_REVIEWED_CURRENT_DEPLOYMENT_ID}" \
  --confirm-promote

After successful argument parsing, success exits 0 and prints a topology-free summary. Validation/integrity failure exits 4, writes no promotion state, prints the stable error code and safe action to stdout, and leaves stderr empty. Argparse usage errors exit 2, write usage to stderr, and do not run promotion. On a first promotion there is no previous deployment; failed integrity verification leaves current absent.

Use --format json for automation. It includes the exact failing local path in errors[0].path, while ordinary text output intentionally omits cache topology. The command is still a promotion attempt, not a read-only doctor: after an operator repairs the cache, a repeated invocation can promote it.

Verification order¶

CI/service identity
  -> acquire the cache-root inter-process promotion lock
  -> validate canonical deployment layout
  -> recompute deployment_id and compare deployment/index runtime fields
  -> read the confined pinned release-set.json
  -> recompute release_id
  -> match release/index artifacts and verify size + SHA-256
       | failure: dpone.error.v1, exit 4, no validation-time mutation
       ` success
  -> copy candidate to activations/<environment>/<deployment_id>
  -> seal activation and pinned release trees read-only
  -> compare the complete candidate/staging tree and revalidate sealed bytes
       | unreadable subtree/non-regular entry: fail closed
  -> exact-create-or-compare any existing activation
  -> prepare a unique relative current symlink to the activation
  -> durably replace current-pointer.json
  -> append the promotion authorization to current-pointer-audit.jsonl
  -> atomically replace current as the final activation commit point
  -> current-pointer result, exit 0

Promotion and recovery serialize through .promotion.lock; snapshot creation, the CAS check, and activation happen while that lock is held. current is always a canonical relative symlink to a sealed activation, never to the mutable build candidate, and is switched last with an atomic local rename. Late edits to the candidate therefore cannot alter what Airflow parses. The pinned release tree is sealed before the snapshot is revalidated. Pointer JSON uses a unique, exclusive, fsynced temporary file. Lock, pointer, and audit files reject symlinks; audit reads and appends use no-follow descriptors. Release and projection files are opened component by component from an anchored cache-root descriptor, so replacing an intermediate directory with a symlink cannot redirect verification outside the cache.

The write steps remain ordered rather than one multi-file transaction. A prepare_current failure leaves pointer, audit, and active current unchanged, but the content-addressed activation may already exist and the pinned release may already be sealed read-only. JSON therefore reports state_may_have_changed: true and recovery_required: false. A pointer preparation failure leaves active current unchanged, but a failure after pointer rename and before directory durability confirmation means the pointer may already be visible. An audit or final activation failure can leave prepared pointer/audit metadata ahead of active current; JSON therefore reports failed_step, state_may_have_changed, and recovery_required conservatively. Do not retry blindly when recovery is required. Run cache-recovery-plan, repair the reported state, and only then retry promotion. The planner independently validates active current and every candidate, including deployment/release fingerprints, the Airflow index, confinement, sizes, and checksums. Invalid candidates are never offered. It reports pointer state as current_deployment_id and active state as current_path_deployment_id. A malformed identity is reported as null plus a structured issue, so recovery JSON remains schema-valid. Pointer release_id must equal the fully validated active projection before audit-only repair can append the existing pointer without changing its original publisher provenance. Malformed historical lines remain warnings; recovery never silently deletes audit history.

All cache identities are lowercase sha256:<64 hex>. The current pointer is trusted only when it also has a non-empty promoted_by and an offset-aware promoted_at; recovery, retention, and safe-sample reuse the same pure pointer contract. Uppercase identities and incomplete authorization fail before control-state mutation.

Recover partial writes¶

First build a read-only plan:

: "${DPONE_SCHEDULER_CACHE_ROOT:?set the absolute scheduler cache root}"

dpone airflow cache-recovery-plan \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --environment prod \
  --format json

The plan command exits 0 when it can produce diagnostics, including when status is blocked; automation must inspect status, issues, and preferred_repair_deployment_id, not only passed or the process exit code. Apply only the reviewed complete candidate:

: "${DPONE_REVIEWED_REPAIR_DEPLOYMENT_ID:?set the repair candidate sha256 digest}"
: "${DPONE_REVIEWED_CURRENT_PATH_DEPLOYMENT_ID:?set the reviewed current-path sha256 digest}"
: "${DPONE_SCHEDULER_CACHE_ROOT:?set the absolute scheduler cache root}"

dpone airflow cache-recovery-apply \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --environment prod \
  --deployment-id "${DPONE_REVIEWED_REPAIR_DEPLOYMENT_ID}" \
  --promoted-by ci://your-platform/dpone-airflow-recovery \
  --allowed-promoter ci://your-platform/dpone-airflow-recovery \
  --expected-current-deployment-id "${DPONE_REVIEWED_CURRENT_PATH_DEPLOYMENT_ID}" \
  --confirm-repair

Use --expect-current-absent instead of --expected-current-deployment-id only when the reviewed plan confirms that the current path is physically absent and the status is repairable. If current exists but has no canonical identity, the plan reports DPONE_CURRENT_PATH_ID_UNAVAILABLE with status: blocked; repair that path under platform change control and rerun the plan. Apply re-plans immediately, rejects a stale active-state guard, rejects candidates absent from the fresh validated plan, and refuses to mutate a healthy cache. Recovery apply and retention apply hold the same cache transaction lock as promotion from fresh plan through mutation, so GC cannot delete a deployment that becomes current concurrently.

Retention apply is also a cache mutation and requires the same explicit actor policy as promotion and recovery. Always run the plan with all evidence and explicit protection inputs first, review it, and copy its generated action. The plan resolves evidence files to effective deployment ids; every occurrence of deployment_id must be canonical. One malformed value invalidates the whole evidence file even when another value is valid. Its action preserves every one as --protect-deployment-id, so apply cannot silently lose the reviewed protection scope:

: "${DPONE_SCHEDULER_CACHE_ROOT:?set the absolute scheduler cache root}"

dpone airflow cache-retention-plan \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --environment prod \
  --evidence-file evidence/run-evidence.json

Example generated action:

: "${DPONE_REVIEWED_PROTECTED_DEPLOYMENT_ID:?set the protected sha256 digest}"
: "${DPONE_SCHEDULER_CACHE_ROOT:?set the absolute scheduler cache root}"

dpone airflow cache-retention-apply \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --environment prod \
  --promoted-by ci://your-platform/dpone-airflow-retention \
  --allowed-promoter ci://your-platform/dpone-airflow-retention \
  --protect-deployment-id "${DPONE_REVIEWED_PROTECTED_DEPLOYMENT_ID}" \
  --confirm-delete

If pointer and physical current state disagree, retention fails with DPONE_DEPLOYMENT_CACHE_RECOVERY_REQUIRED before deleting any candidate. The active projection itself must also pass full deployment/index/release/artifact validation under that lock; fingerprint or artifact corruption requires recovery and cannot turn another valid deployment into a delete candidate. Each delete candidate passes the same validation. The entire delete set is validated before the first removal. If a later filesystem deletion fails, JSON returns DPONE_DEPLOYMENT_CACHE_GC_DELETE_FAILED with failed_step, state_may_have_changed, deleted_deployment_ids, and failed_deployment_id; rerun the plan before any retry. An invalid directory name has no trustworthy deployment identity, so its quarantine item carries deployment_id: null with the exact error code and path; it is never included in delete or skipped-ID lists. Human-readable plan and apply output reports NEEDS_ATTENTION, labels the item as an unidentified cache entry, and shows its stable error code without exposing the cache path. JSON remains the topology-bearing diagnostic format.

The retention schemas permit a null deployment identity only for an incomplete/invalid quarantine item and, in apply output, only with action: skipped. protect, delete, and deleted actions always require a canonical deployment digest. This prevents a damaged cache entry from being misrepresented as a destructive action.

Apply exits 0 after a complete repair, 4 when confirmation is missing, and 1 for other recovery failures. If apply itself reports DPONE_CACHE_PROMOTION_WRITE_FAILED, local state may have changed; run the plan again before any retry.

Recovery by error family¶

Error family	Meaning	Recovery
`DPONE_AIRFLOW_INDEX_NOT_FOUND`, `DPONE_AIRFLOW_INDEX_READ_FAILED`, `DPONE_AIRFLOW_INDEX_TOO_LARGE`, `DPONE_AIRFLOW_INDEX_JSON_INVALID`, index schema/field/identity errors	The provider cannot establish a trusted deployment snapshot and returns `fatal: true`.	Run `cache-recovery-plan`. Rematerialize or rebuild the exact projection as directed, then use a fresh reviewed CAS guard; never suppress the generated loader failure.
`DPONE_AIRFLOW_DAG_SPEC_LOAD_FAILED`, `DPONE_CACHE_ARTIFACT_MISSING`, `DPONE_CACHE_ARTIFACT_SIZE_MISMATCH`, `DPONE_CACHE_CHECKSUM_MISMATCH` with `fatal: false`	One listed DAG/pack is corrupt or unreadable; unrelated DAGs may remain available.	Restore the immutable release bytes from trusted storage or build a new release/projection. Do not patch the affected file in place.
`DPONE_RELEASE_NOT_FOUND`	The release is absent or the projection has no usable release reference.	Materialize the pinned release, or rebuild the deployment projection with a valid `release_ref`.
`DPONE_RELEASE_*INVALID`, `DPONE_RELEASE_FINGERPRINT_MISMATCH`	The immutable release contract or content address is wrong.	Build a new immutable release-set, then rebuild its deployment projection.
`DPONE_RELEASE_INDEX_ARTIFACT_MISMATCH`, index/reference errors	The deployment projection does not represent the pinned release.	Rebuild the deployment projection from the unchanged release.
Missing, unreadable, size-mismatched, or checksum-mismatched artifact	Local release content is incomplete or corrupt.	Restore the exact content-addressed release from trusted storage; never edit it in place.
`DPONE_CACHE_ARTIFACT_TOO_LARGE`	The artifact exceeds the configured local verification budget.	Review the platform size policy or build a smaller artifact; restoring the same oversized file is not a fix.
Confirmation required	The explicit mutation acknowledgement is absent.	Review the candidate and add `--confirm-promote`.
Promoter missing/unauthorized	The caller is not an allowed CI/service identity.	Use an allowed identity or update the platform allowlist through review.
`DPONE_CURRENT_POINTER_CAS_MISMATCH`	Another promotion or recovery changed active current.	Refresh the plan and retry with its active deployment ID as the CAS guard.
`DPONE_CURRENT_RELEASE_ID_MISMATCH`	Pointer release does not match the validated active projection.	Run confirmed recovery for the reviewed deployment; audit-only repair is forbidden.
`DPONE_CURRENT_PATH_ID_UNAVAILABLE`, `DPONE_DEPLOYMENT_CACHE_RECOVERY_BLOCKED`	`current` exists but has no canonical identity for stale-plan CAS.	Repair or remove the unsafe path through platform change control, rerun the plan, then apply only a fresh repairable plan.
`DPONE_DEPLOYMENT_FINGERPRINT_MISMATCH`, mirror mismatch	Deployment content or its Airflow projection no longer matches `deployment_id`.	Rebuild the immutable deployment projection; never rebless the old ID.
`DPONE_DEPLOYMENT_CACHE_RECOVERY_REQUIRED`	Existing pointer and active current are inconsistent.	Stop normal promotion and run the recovery plan/apply workflow.
`DPONE_CACHE_PROMOTION_LOCK_FAILED`	The exclusive control-plane lock is unsafe or unavailable.	Repair cache-root ownership/type; never replace it with a symlink.
`DPONE_DEPLOYMENT_PATH_OUTSIDE_CACHE_ROOT`	The selected projection is outside the configured cache.	Select or materialize the deployment below the configured cache root.
`DPONE_DEPLOYMENT_PATH_INVALID`	The selected path does not use the canonical deployment layout.	Use `deployments/<environment>/<deployment_id>` below the cache root.
`DPONE_DEPLOYMENT_ACTIVATION_FAILED`	A sealed activation snapshot could not be copied, verified, or published.	Keep the existing current deployment, repair local cache permissions or storage, then retry promotion.
`DPONE_DEPLOYMENT_ACTIVATION_CONFLICT`	Existing immutable activation bytes differ from the newly validated candidate for the same deployment ID.	Quarantine the cache and rematerialize from trusted content; never overwrite the activation in place.
`DPONE_DEPLOYMENT_CACHE_GC_VALIDATION_FAILED`	At least one reviewed delete candidate could not be revalidated before mutation. JSON includes the underlying stable `cause_code`.	Repair or rematerialize the candidate, then generate and review a fresh retention plan. No deployment was deleted by this apply.
`DPONE_DEPLOYMENT_CACHE_GC_DELETE_FAILED`	Filesystem deletion stopped after zero or more reviewed candidates.	Inspect `deleted_deployment_ids` and `failed_deployment_id`, repair the filesystem, then rerun the plan; never replay the stale apply blindly.
Incomplete/environment/schema deployment errors	The projection is not promotable as requested.	Rebuild/select the complete projection for the requested environment.
`DPONE_CACHE_PROMOTION_WRITE_FAILED`	A local current, pointer, or audit write failed after validation.	Run `cache-recovery-plan`; reconcile split state before retry.

For checksum-specific fields and output examples, see DPONE_CACHE_CHECKSUM_MISMATCH. After a successful retry, verify local pointer health:

: "${DPONE_SCHEDULER_CACHE_ROOT:?set the absolute scheduler cache root}"

dpone airflow cache-recovery-plan \
  --cache-root "${DPONE_SCHEDULER_CACHE_ROOT}" \
  --environment prod

Python integration¶

The CLI is the recommended platform surface. An embedded materializer can use the same public runtime service and structured exception:

from pathlib import Path

from dpone.runtime.deployment_cache import (
    DeploymentCacheError,
    DeploymentCacheMaterializer,
)

cache_root = Path("/opt/airflow/dags/.dpone-cache")
deployment_id = "sha256:" + "a" * 64
deployment_dir = cache_root / "deployments" / "prod" / deployment_id.replace(":", "-")
materializer = DeploymentCacheMaterializer(
    cache_root,
    allowed_promoters=("ci://your-platform/dpone-airflow",),
    max_artifact_bytes=64 * 1024 * 1024,
)
try:
    current = materializer.promote(
        deployment_dir,
        environment="prod",
        promoted_by="ci://your-platform/dpone-airflow",
        expect_current_absent=True,
    )
except DeploymentCacheError as exc:
    # exc.code and exc.path are stable inputs for platform error handling.
    raise

The service performs local reads and writes only. Callers must still provide the trusted release handoff, explicit promoter policy, and recovery operation.

Continue with the Airflow self-service architecture, the approved cache promotion feature design, the generated cache-sync CLI reference, or the GitOps schema catalog.